[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:02:00 dist-test-slave-7zdl krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:02:00 dist-test-slave-7zdl krb5kdc[399](info): set up 2 sockets
Sep 29 01:02:00 dist-test-slave-7zdl krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 01:02:01.955343 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:02:02 dist-test-slave-7zdl krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:02:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:02:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:02:02.073863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41137
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:02.180226 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:02.180514 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:02.180590 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:02.184624 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:02:02.184718 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:02.184756 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:02.184810 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:02:02.184849 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:02:02.190042 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:02.191109 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:02.192139 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:02:02.200905 412 server_base.cc:1034] running on GCE node
W20260929 01:02:02.200929 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:02.200968 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:02.201172 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:02.201709 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:02.202838 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:02.204015 412 hybrid_clock.cc:648] HybridClock initialized: now 1790643722203964 us; error 61 us; skew 500 ppm
Sep 29 01:02:02 dist-test-slave-7zdl krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:02.206641 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:02:02.207497 412 webserver.cc:460] Webserver started at http://127.0.98.62:37901/ using document root <none> and password file <none>
I20260929 01:02:02.208032 412 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:02.208118 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:02.208339 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:02.210314 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "8486c88f042a46fda54c0ec17c3b0c04"
format_stamp: "Formatted at 2026-09-29 01:02:02 on dist-test-slave-7zdl"
server_key: "a20c9580df2031fd6e6919dfb319171f"
server_key_iv: "0541572c084f2931cff542090e2033b7"
server_key_version: "encryptionkey@0"
I20260929 01:02:02.210919 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "8486c88f042a46fda54c0ec17c3b0c04"
format_stamp: "Formatted at 2026-09-29 01:02:02 on dist-test-slave-7zdl"
server_key: "a20c9580df2031fd6e6919dfb319171f"
server_key_iv: "0541572c084f2931cff542090e2033b7"
server_key_version: "encryptionkey@0"
I20260929 01:02:02.214485 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 01:02:02.216799 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:02.217983 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 01:02:02.218137 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "8486c88f042a46fda54c0ec17c3b0c04"
format_stamp: "Formatted at 2026-09-29 01:02:02 on dist-test-slave-7zdl"
server_key: "a20c9580df2031fd6e6919dfb319171f"
server_key_iv: "0541572c084f2931cff542090e2033b7"
server_key_version: "encryptionkey@0"
I20260929 01:02:02.218245 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:02.234122 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:02.236970 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:02.237188 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:02.244217 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41137
I20260929 01:02:02.244243 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41137 every 8 connection(s)
I20260929 01:02:02.245124 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:02:02.247731 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:02.249711 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 412
I20260929 01:02:02.249851 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:02:02.251175 392 external_mini_cluster.cc:1428] Setting key 8826bfaaf50a1bd7444333f599333d35
I20260929 01:02:02.253232 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8486c88f042a46fda54c0ec17c3b0c04: Bootstrap starting.
I20260929 01:02:02.255808 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8486c88f042a46fda54c0ec17c3b0c04: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:02.256493 480 log.cc:824] T 00000000000000000000000000000000 P 8486c88f042a46fda54c0ec17c3b0c04: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:02.258144 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8486c88f042a46fda54c0ec17c3b0c04: No bootstrap required, opened a new log
I20260929 01:02:02.261440 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8486c88f042a46fda54c0ec17c3b0c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8486c88f042a46fda54c0ec17c3b0c04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41137 } }
I20260929 01:02:02.261683 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8486c88f042a46fda54c0ec17c3b0c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:02.261762 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8486c88f042a46fda54c0ec17c3b0c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8486c88f042a46fda54c0ec17c3b0c04, State: Initialized, Role: FOLLOWER
I20260929 01:02:02.262214 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8486c88f042a46fda54c0ec17c3b0c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8486c88f042a46fda54c0ec17c3b0c04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41137 } }
I20260929 01:02:02.262362 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8486c88f042a46fda54c0ec17c3b0c04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:02:02.262447 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8486c88f042a46fda54c0ec17c3b0c04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:02:02.262568 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8486c88f042a46fda54c0ec17c3b0c04 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:02.263475 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8486c88f042a46fda54c0ec17c3b0c04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8486c88f042a46fda54c0ec17c3b0c04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41137 } }
I20260929 01:02:02.263813 480 leader_election.cc:302] T 00000000000000000000000000000000 P 8486c88f042a46fda54c0ec17c3b0c04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8486c88f042a46fda54c0ec17c3b0c04; no voters:
I20260929 01:02:02.264155 480 leader_election.cc:288] T 00000000000000000000000000000000 P 8486c88f042a46fda54c0ec17c3b0c04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:02:02.264230 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8486c88f042a46fda54c0ec17c3b0c04 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:02.264474 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8486c88f042a46fda54c0ec17c3b0c04 [term 1 LEADER]: Becoming Leader. State: Replica: 8486c88f042a46fda54c0ec17c3b0c04, State: Running, Role: LEADER
I20260929 01:02:02.264806 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8486c88f042a46fda54c0ec17c3b0c04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8486c88f042a46fda54c0ec17c3b0c04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41137 } }
I20260929 01:02:02.265373 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8486c88f042a46fda54c0ec17c3b0c04 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:02:02.266412 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8486c88f042a46fda54c0ec17c3b0c04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8486c88f042a46fda54c0ec17c3b0c04. Latest consensus state: current_term: 1 leader_uuid: "8486c88f042a46fda54c0ec17c3b0c04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8486c88f042a46fda54c0ec17c3b0c04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41137 } } }
I20260929 01:02:02.266561 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8486c88f042a46fda54c0ec17c3b0c04 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:02.266721 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8486c88f042a46fda54c0ec17c3b0c04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8486c88f042a46fda54c0ec17c3b0c04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8486c88f042a46fda54c0ec17c3b0c04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41137 } } }
I20260929 01:02:02.266934 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8486c88f042a46fda54c0ec17c3b0c04 [sys.catalog]: This master's current role is: LEADER
Sep 29 01:02:02 dist-test-slave-7zdl krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260929 01:02:02.270349 500 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8486c88f042a46fda54c0ec17c3b0c04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:02:02.270457 500 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:02:02.270563 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:02:02.271313 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:02:02.275115 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:02.254033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33398 (local address 127.0.98.62:41137)
0929 01:02:02.254421 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:02.254435 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:02.255540 (+ 1105us) server_negotiation.cc:407] Connection header received
0929 01:02:02.258502 (+ 2962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:02.258523 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:02.258871 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:02.259175 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:02.261212 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:02.262331 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:02.263650 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:02.264167 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:02.264342 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:02.269231 (+ 4889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:02.269237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:02.269253 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:02.269282 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:02.271358 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:02.271982 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:02.271988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:02.271993 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:02.272133 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:02.272529 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:02.272533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:02.272535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:02.273029 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:02.273212 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:02.273985 (+ 773us) server_negotiation.cc:299] Negotiation successful
0929 01:02:02.274270 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":126,"threads_started":1}
I20260929 01:02:02.277271 494 catalog_manager.cc:1347] Generated new cluster ID: 58e1220343bc4e3e9d1049b39b063b22
I20260929 01:02:02.277375 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:02:02.306939 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:02:02.307925 494 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:02:02.312686 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8486c88f042a46fda54c0ec17c3b0c04: Generated new TSK 0
I20260929 01:02:02.313459 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 01:02:02.349272 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41137
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:02.456476 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:02.456723 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:02.456765 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:02.460744 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:02.460803 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:02.460911 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:02:02.466248 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41137
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:02.467355 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:02.468416 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:02.477789 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:02.477846 506 server_base.cc:1034] running on GCE node
W20260929 01:02:02.477801 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:02.477797 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:02.478482 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:02.479151 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:02.480306 506 hybrid_clock.cc:648] HybridClock initialized: now 1790643722480276 us; error 44 us; skew 500 ppm
Sep 29 01:02:02 dist-test-slave-7zdl krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:02.483089 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:02:02.483945 506 webserver.cc:460] Webserver started at http://127.0.98.1:40887/ using document root <none> and password file <none>
I20260929 01:02:02.484503 506 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:02.484565 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:02.484781 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:02.486745 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "435e5dc50d0c4ffaa5e5df019af4a5ba"
format_stamp: "Formatted at 2026-09-29 01:02:02 on dist-test-slave-7zdl"
server_key: "6df4e390e432731a244879cce49c3a16"
server_key_iv: "4c36a556633b7082672089cffb448651"
server_key_version: "encryptionkey@0"
I20260929 01:02:02.487352 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "435e5dc50d0c4ffaa5e5df019af4a5ba"
format_stamp: "Formatted at 2026-09-29 01:02:02 on dist-test-slave-7zdl"
server_key: "6df4e390e432731a244879cce49c3a16"
server_key_iv: "4c36a556633b7082672089cffb448651"
server_key_version: "encryptionkey@0"
I20260929 01:02:02.490917 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:02:02.493166 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:02.494263 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:02.494381 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "435e5dc50d0c4ffaa5e5df019af4a5ba"
format_stamp: "Formatted at 2026-09-29 01:02:02 on dist-test-slave-7zdl"
server_key: "6df4e390e432731a244879cce49c3a16"
server_key_iv: "4c36a556633b7082672089cffb448651"
server_key_version: "encryptionkey@0"
I20260929 01:02:02.494477 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:02.511461 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:02.514637 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:02.514854 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:02.515417 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:02.516181 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:02.516227 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:02.516295 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:02.516326 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:02.524930 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46059
I20260929 01:02:02.524962 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46059 every 8 connection(s)
I20260929 01:02:02.525991 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:02:02.534945 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 506
I20260929 01:02:02.535063 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:02:02.535291 392 external_mini_cluster.cc:1428] Setting key 47dec9bace1859300e6253e6ceb6103c
Sep 29 01:02:02 dist-test-slave-7zdl krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:02.539608 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:02.529486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33049 (local address 127.0.98.62:41137)
0929 01:02:02.529611 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:02.529615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:02.529636 (+ 21us) server_negotiation.cc:407] Connection header received
0929 01:02:02.530494 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:02.530497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:02.530548 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:02.530619 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:02.531752 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:02.532266 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:02.533304 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:02.533527 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:02.533598 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:02.535987 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:02.535990 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:02.535992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:02.536013 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:02.537452 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:02.538055 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:02.538058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:02.538059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:02.538135 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:02.538522 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:02.538526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:02.538528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:02.538708 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:02.538840 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:02.539307 (+ 467us) server_negotiation.cc:299] Negotiation successful
0929 01:02:02.539430 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 01:02:02.540359 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:02.528469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41137 (local address 127.0.98.1:33049)
0929 01:02:02.529108 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:02.529161 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:02.530324 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:02.530741 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:02.530747 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:02.531150 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:02.531632 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:02.531645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:02.532367 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:02.532369 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:02.533210 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:02.533218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:02.533645 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:02.533648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:02.533781 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:02.534256 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:02.534271 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:02.535817 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:02.537568 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:02.537577 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:02.537593 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:02.537956 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:02.538223 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:02.538228 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:02.538233 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:02.538424 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:02.538829 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:02.538835 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:02.539077 (+ 242us) client_negotiation.cc:769] Sending connection context
0929 01:02:02.539255 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 01:02:02.539558 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":145,"threads_started":1}
I20260929 01:02:02.541563 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41137
I20260929 01:02:02.541792 635 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:02.542238 635 heartbeater.cc:511] Master 127.0.98.62:41137 requested a full tablet report, sending...
I20260929 01:02:02.543903 444 ts_manager.cc:194] Registered new tserver with Master: 435e5dc50d0c4ffaa5e5df019af4a5ba (127.0.98.1:46059)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 01:02:02.545256 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 01:02:02.562458 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41137
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:02.670420 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:02.670681 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:02.670723 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:02.674754 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:02.674871 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:02.674988 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:02:02.680177 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41137
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:02.681144 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:02.682118 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:02.691650 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:02.691650 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:02.692000 640 server_base.cc:1034] running on GCE node
W20260929 01:02:02.691676 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:02.692379 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:02.692997 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:02.694172 640 hybrid_clock.cc:648] HybridClock initialized: now 1790643722694143 us; error 42 us; skew 500 ppm
Sep 29 01:02:02 dist-test-slave-7zdl krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:02.696882 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:02:02.697676 640 webserver.cc:460] Webserver started at http://127.0.98.2:45611/ using document root <none> and password file <none>
I20260929 01:02:02.698216 640 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:02.698274 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:02.698429 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:02.700361 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "894f53b3292d4a39910a8d217fef058c"
format_stamp: "Formatted at 2026-09-29 01:02:02 on dist-test-slave-7zdl"
server_key: "711c337850f234f8568eb39ce27f7ce1"
server_key_iv: "82ae528b67f6534fef72abd31b5d6fe5"
server_key_version: "encryptionkey@0"
I20260929 01:02:02.700843 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "894f53b3292d4a39910a8d217fef058c"
format_stamp: "Formatted at 2026-09-29 01:02:02 on dist-test-slave-7zdl"
server_key: "711c337850f234f8568eb39ce27f7ce1"
server_key_iv: "82ae528b67f6534fef72abd31b5d6fe5"
server_key_version: "encryptionkey@0"
I20260929 01:02:02.704262 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 01:02:02.706686 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:02.707851 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:02.707971 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "894f53b3292d4a39910a8d217fef058c"
format_stamp: "Formatted at 2026-09-29 01:02:02 on dist-test-slave-7zdl"
server_key: "711c337850f234f8568eb39ce27f7ce1"
server_key_iv: "82ae528b67f6534fef72abd31b5d6fe5"
server_key_version: "encryptionkey@0"
I20260929 01:02:02.708066 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:02.734247 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:02.736919 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:02.737112 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:02.737699 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:02.738471 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:02.738516 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:02.738583 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:02.738623 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:02.748066 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32899
I20260929 01:02:02.748096 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32899 every 8 connection(s)
I20260929 01:02:02.748988 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 01:02:02 dist-test-slave-7zdl krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:02.758061 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 640
I20260929 01:02:02.758196 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:02:02.758455 392 external_mini_cluster.cc:1428] Setting key 5b3619527ad81ed27ca499b6c85556cb
I20260929 01:02:02.762089 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:02.750614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51183 (local address 127.0.98.62:41137)
0929 01:02:02.750740 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:02.750743 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:02.751589 (+ 846us) server_negotiation.cc:407] Connection header received
0929 01:02:02.752850 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:02.752853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:02.752899 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:02.752971 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:02.754034 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:02.754579 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:02.755707 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:02.755916 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:02.755994 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:02.758435 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:02.758438 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:02.758440 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:02.758463 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:02.760146 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:02.760633 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:02.760638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:02.760640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:02.760724 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:02.761050 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:02.761055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:02.761056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:02.761234 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:02.761338 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:02.761804 (+ 466us) server_negotiation.cc:299] Negotiation successful
0929 01:02:02.761915 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 01:02:02.762869 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:02.751004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41137 (local address 127.0.98.2:51183)
0929 01:02:02.751409 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:02.751448 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:02.752662 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:02.753120 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:02.753126 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:02.753450 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:02.753914 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:02.753926 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:02.754677 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:02.754679 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:02.755605 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:02.755613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:02.756034 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:02.756036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:02.756163 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:02.756748 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:02.756763 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:02.758267 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:02.760260 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:02.760265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:02.760277 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:02.760523 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:02.760818 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:02.760821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:02.760823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:02.760954 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:02.761336 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:02.761342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:02.761568 (+ 226us) client_negotiation.cc:769] Sending connection context
0929 01:02:02.761762 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 01:02:02.762052 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":106,"threads_started":1}
I20260929 01:02:02.763967 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41137
I20260929 01:02:02.764221 769 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:02.764777 769 heartbeater.cc:511] Master 127.0.98.62:41137 requested a full tablet report, sending...
I20260929 01:02:02.765856 444 ts_manager.cc:194] Registered new tserver with Master: 894f53b3292d4a39910a8d217fef058c (127.0.98.2:32899)
I20260929 01:02:02.766718 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51183
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 01:02:02.784186 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41137
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:02.893957 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:02.894192 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:02.894234 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:02.898139 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:02.898190 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:02.898247 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:02:02.903342 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41137
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:02.904280 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:02.905192 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:02.914659 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:02.914675 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:02.914729 774 server_base.cc:1034] running on GCE node
W20260929 01:02:02.914659 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:02.915370 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:02.915968 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:02.917153 774 hybrid_clock.cc:648] HybridClock initialized: now 1790643722917138 us; error 36 us; skew 500 ppm
Sep 29 01:02:02 dist-test-slave-7zdl krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:02.919745 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:02:02.920558 774 webserver.cc:460] Webserver started at http://127.0.98.3:43985/ using document root <none> and password file <none>
I20260929 01:02:02.921105 774 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:02.921226 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:02.921468 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:02.923580 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "d1cc4bcb7a8a4f518053a103f56e369c"
format_stamp: "Formatted at 2026-09-29 01:02:02 on dist-test-slave-7zdl"
server_key: "167884101a5273e7dac17cef80f4180f"
server_key_iv: "944dcc53de92e90852c7d8abdb58d4de"
server_key_version: "encryptionkey@0"
I20260929 01:02:02.924150 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d1cc4bcb7a8a4f518053a103f56e369c"
format_stamp: "Formatted at 2026-09-29 01:02:02 on dist-test-slave-7zdl"
server_key: "167884101a5273e7dac17cef80f4180f"
server_key_iv: "944dcc53de92e90852c7d8abdb58d4de"
server_key_version: "encryptionkey@0"
I20260929 01:02:02.927717 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:02:02.930080 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:02.931249 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:02:02.931375 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "d1cc4bcb7a8a4f518053a103f56e369c"
format_stamp: "Formatted at 2026-09-29 01:02:02 on dist-test-slave-7zdl"
server_key: "167884101a5273e7dac17cef80f4180f"
server_key_iv: "944dcc53de92e90852c7d8abdb58d4de"
server_key_version: "encryptionkey@0"
I20260929 01:02:02.931479 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:02.941167 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:02.943678 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:02.943861 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:02.944399 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:02.945145 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:02.945215 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:02.945279 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:02.945329 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:02.955067 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45773
I20260929 01:02:02.955087 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45773 every 8 connection(s)
I20260929 01:02:02.955885 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:02:02.959466 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 774
I20260929 01:02:02.959591 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:02:02.959860 392 external_mini_cluster.cc:1428] Setting key 3c52ae3a307859cdf0eb56c5aade3225
Sep 29 01:02:02 dist-test-slave-7zdl krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:02.968998 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:02.957553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49349 (local address 127.0.98.62:41137)
0929 01:02:02.957708 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:02.957713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:02.958454 (+ 741us) server_negotiation.cc:407] Connection header received
0929 01:02:02.959854 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:02.959858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:02.959919 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:02.960013 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:02.961229 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:02.961969 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:02.963114 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:02.963343 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:02.963449 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:02.965867 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:02.965871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:02.965874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:02.965903 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:02.967366 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:02.967817 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:02.967820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:02.967821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:02.967890 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:02.968164 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:02.968166 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:02.968167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:02.968318 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:02.968389 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:02.968798 (+ 409us) server_negotiation.cc:299] Negotiation successful
0929 01:02:02.968885 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 01:02:02.969738 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:02.957905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41137 (local address 127.0.98.3:49349)
0929 01:02:02.958317 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:02.958355 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:02.959614 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:02.960152 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:02.960158 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:02.960545 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:02.961092 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:02.961106 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:02.962091 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:02.962096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:02.962994 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:02.963003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:02.963479 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:02.963483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:02.963647 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:02.964240 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:02.964255 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:02.965705 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:02.967463 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:02.967467 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:02.967477 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:02.967736 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:02.967967 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:02.967969 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:02.967970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:02.968091 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:02.968398 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:02.968402 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:02.968615 (+ 213us) client_negotiation.cc:769] Sending connection context
0929 01:02:02.968758 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 01:02:02.968981 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":113,"threads_started":1}
I20260929 01:02:02.970710 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41137
I20260929 01:02:02.970945 903 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:02.971381 903 heartbeater.cc:511] Master 127.0.98.62:41137 requested a full tablet report, sending...
I20260929 01:02:02.972313 444 ts_manager.cc:194] Registered new tserver with Master: d1cc4bcb7a8a4f518053a103f56e369c (127.0.98.3:45773)
I20260929 01:02:02.972846 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49349
I20260929 01:02:02.973920 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 01:02:02 dist-test-slave-7zdl krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643722, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 01:02:02 dist-test-slave-7zdl krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643722, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 01:02:02.998838 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:02.989558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60004 (local address 127.0.98.1:46059)
0929 01:02:02.989932 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:02.989950 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:02.989970 (+ 20us) server_negotiation.cc:407] Connection header received
0929 01:02:02.990018 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:02.990022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:02.990196 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:02.990347 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:02.991062 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:02.991868 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:02.992739 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:02.993107 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:02.993222 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:02.995528 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:02.995536 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:02.995547 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:02.995571 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:02.996984 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:02.997406 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:02.997411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:02.997412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:02.997493 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:02.997768 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:02.997773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:02.997774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:02.998086 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:02.998297 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:02.998464 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 01:02:02.998625 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":125,"threads_started":1}
W20260929 01:02:02.999799 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60004
Sep 29 01:02:03 dist-test-slave-7zdl krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 01:02:03 dist-test-slave-7zdl krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 01:02:03.023617 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:03.016025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60010 (local address 127.0.98.1:46059)
0929 01:02:03.016173 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:03.016177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:03.016229 (+ 52us) server_negotiation.cc:407] Connection header received
0929 01:02:03.016330 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:03.016332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:03.016370 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:03.016437 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:03.017083 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:03.017587 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:03.018381 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:03.018600 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:03.018671 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:03.020787 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:03.020790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:03.020792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:03.020812 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:03.022115 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:03.022519 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:03.022521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:03.022522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:03.022609 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:03.022903 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:03.022905 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:03.022906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:03.023047 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:03.023143 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:03.023313 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 01:02:03.023399 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 01:02:03.024574 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 506
I20260929 01:02:03.030419 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 640
I20260929 01:02:03.037772 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 774
I20260929 01:02:03.042627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 412
2026-09-29T01:02:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2149 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:02:03 dist-test-slave-7zdl krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:02:03 dist-test-slave-7zdl krb5kdc[917](info): set up 2 sockets
Sep 29 01:02:03 dist-test-slave-7zdl krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260929 01:02:04.086476 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:02:04 dist-test-slave-7zdl krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:02:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:02:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:02:04.203749 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40747
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:04.309162 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:04.309440 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:04.309511 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:04.313542 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:02:04.313622 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:04.313663 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:04.313719 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:02:04.313755 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:02:04.318864 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:04.319828 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:04.320780 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:04.328871 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:04.328879 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:04.329154 930 server_base.cc:1034] running on GCE node
W20260929 01:02:04.328919 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:04.329671 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:04.330732 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:04.331915 930 hybrid_clock.cc:648] HybridClock initialized: now 1790643724331903 us; error 23 us; skew 500 ppm
Sep 29 01:02:04 dist-test-slave-7zdl krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:04.337808 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:02:04.338685 930 webserver.cc:460] Webserver started at http://127.0.98.62:42497/ using document root <none> and password file <none>
I20260929 01:02:04.339272 930 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:04.339357 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:04.339584 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:04.341538 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "ef2b1953f03540d6bd5033e09de8041f"
format_stamp: "Formatted at 2026-09-29 01:02:04 on dist-test-slave-7zdl"
server_key: "3e550119894f8b3b4b5a4b9e30760155"
server_key_iv: "10558b4cb3837d8d2ae42734e353d3e2"
server_key_version: "encryptionkey@0"
I20260929 01:02:04.342074 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "ef2b1953f03540d6bd5033e09de8041f"
format_stamp: "Formatted at 2026-09-29 01:02:04 on dist-test-slave-7zdl"
server_key: "3e550119894f8b3b4b5a4b9e30760155"
server_key_iv: "10558b4cb3837d8d2ae42734e353d3e2"
server_key_version: "encryptionkey@0"
I20260929 01:02:04.345573 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 01:02:04.347829 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:04.348944 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:02:04.349100 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "ef2b1953f03540d6bd5033e09de8041f"
format_stamp: "Formatted at 2026-09-29 01:02:04 on dist-test-slave-7zdl"
server_key: "3e550119894f8b3b4b5a4b9e30760155"
server_key_iv: "10558b4cb3837d8d2ae42734e353d3e2"
server_key_version: "encryptionkey@0"
I20260929 01:02:04.349206 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:04.363713 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:04.366353 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:04.366544 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:04.373073 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40747
I20260929 01:02:04.373140 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40747 every 8 connection(s)
I20260929 01:02:04.373939 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:02:04.376647 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:04.379341 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 930
I20260929 01:02:04.379454 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:02:04.379699 392 external_mini_cluster.cc:1428] Setting key 147f2b33a365a111617061b41a5c2b7f
I20260929 01:02:04.382266 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef2b1953f03540d6bd5033e09de8041f: Bootstrap starting.
I20260929 01:02:04.384706 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef2b1953f03540d6bd5033e09de8041f: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:04.385526 998 log.cc:824] T 00000000000000000000000000000000 P ef2b1953f03540d6bd5033e09de8041f: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:04.387511 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef2b1953f03540d6bd5033e09de8041f: No bootstrap required, opened a new log
Sep 29 01:02:04 dist-test-slave-7zdl krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:04.391443 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef2b1953f03540d6bd5033e09de8041f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2b1953f03540d6bd5033e09de8041f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40747 } }
I20260929 01:02:04.391710 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef2b1953f03540d6bd5033e09de8041f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:04.391774 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef2b1953f03540d6bd5033e09de8041f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef2b1953f03540d6bd5033e09de8041f, State: Initialized, Role: FOLLOWER
I20260929 01:02:04.392239 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef2b1953f03540d6bd5033e09de8041f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2b1953f03540d6bd5033e09de8041f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40747 } }
I20260929 01:02:04.392367 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef2b1953f03540d6bd5033e09de8041f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:02:04.392450 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef2b1953f03540d6bd5033e09de8041f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:02:04.392552 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef2b1953f03540d6bd5033e09de8041f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:04.393448 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef2b1953f03540d6bd5033e09de8041f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2b1953f03540d6bd5033e09de8041f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40747 } }
I20260929 01:02:04.393774 998 leader_election.cc:302] T 00000000000000000000000000000000 P ef2b1953f03540d6bd5033e09de8041f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef2b1953f03540d6bd5033e09de8041f; no voters:
I20260929 01:02:04.394044 998 leader_election.cc:288] T 00000000000000000000000000000000 P ef2b1953f03540d6bd5033e09de8041f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:02:04.394151 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef2b1953f03540d6bd5033e09de8041f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:04.394186 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:04.380981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46234 (local address 127.0.98.62:40747)
0929 01:02:04.381364 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:04.381379 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:04.381411 (+ 32us) server_negotiation.cc:407] Connection header received
0929 01:02:04.381979 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:04.381994 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:04.382297 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:04.382610 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:04.383435 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:04.384398 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:04.385577 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:04.386071 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:04.386185 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:04.388645 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:04.388651 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:04.388666 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:04.388695 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:04.391178 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:04.391577 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:04.391582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:04.391587 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:04.391718 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:04.392064 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:04.392069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:04.392071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:04.392541 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:04.392768 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:04.393057 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 01:02:04.393334 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":119,"threads_started":1}
I20260929 01:02:04.394415 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef2b1953f03540d6bd5033e09de8041f [term 1 LEADER]: Becoming Leader. State: Replica: ef2b1953f03540d6bd5033e09de8041f, State: Running, Role: LEADER
I20260929 01:02:04.394699 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef2b1953f03540d6bd5033e09de8041f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2b1953f03540d6bd5033e09de8041f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40747 } }
I20260929 01:02:04.395265 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef2b1953f03540d6bd5033e09de8041f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:02:04.395815 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef2b1953f03540d6bd5033e09de8041f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef2b1953f03540d6bd5033e09de8041f. Latest consensus state: current_term: 1 leader_uuid: "ef2b1953f03540d6bd5033e09de8041f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2b1953f03540d6bd5033e09de8041f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40747 } } }
I20260929 01:02:04.395916 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef2b1953f03540d6bd5033e09de8041f [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:04.396111 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef2b1953f03540d6bd5033e09de8041f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef2b1953f03540d6bd5033e09de8041f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2b1953f03540d6bd5033e09de8041f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40747 } } }
I20260929 01:02:04.396190 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef2b1953f03540d6bd5033e09de8041f [sys.catalog]: This master's current role is: LEADER
W20260929 01:02:04.399750 1018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ef2b1953f03540d6bd5033e09de8041f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:02:04.399814 1018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:02:04.399890 1009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:02:04.400805 1009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:02:04.405926 1009 catalog_manager.cc:1347] Generated new cluster ID: bdeabda569394aa7ad2cdd6fcf66cfec
I20260929 01:02:04.405984 1009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:02:04.415273 1009 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:02:04.416409 1009 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:02:04.422108 1009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef2b1953f03540d6bd5033e09de8041f: Generated new TSK 0
I20260929 01:02:04.422865 1009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260929 01:02:05.421057 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.992s user 0.003s sys 0.000s
I20260929 01:02:05.421496 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40747
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:05.537819 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:05.538115 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:05.538206 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:05.543015 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:05.543076 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:05.543192 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:02:05.548594 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40747
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:05.549691 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:05.550733 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:05.560814 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:05.560978 1024 server_base.cc:1034] running on GCE node
W20260929 01:02:05.560876 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:05.560814 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:05.561646 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:05.562335 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:05.563498 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790643725563453 us; error 56 us; skew 500 ppm
Sep 29 01:02:05 dist-test-slave-7zdl krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:05.566394 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:02:05.567306 1024 webserver.cc:460] Webserver started at http://127.0.98.1:40587/ using document root <none> and password file <none>
I20260929 01:02:05.567883 1024 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:05.567974 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:05.568220 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:05.570250 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "df2edc90c1d545d3be404c3f11954055"
format_stamp: "Formatted at 2026-09-29 01:02:05 on dist-test-slave-7zdl"
server_key: "676a076f07d0bd025bc82ea9781e87d6"
server_key_iv: "2fb2707a5408214928767cdd2e021779"
server_key_version: "encryptionkey@0"
I20260929 01:02:05.570855 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "df2edc90c1d545d3be404c3f11954055"
format_stamp: "Formatted at 2026-09-29 01:02:05 on dist-test-slave-7zdl"
server_key: "676a076f07d0bd025bc82ea9781e87d6"
server_key_iv: "2fb2707a5408214928767cdd2e021779"
server_key_version: "encryptionkey@0"
I20260929 01:02:05.574447 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:02:05.576869 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:05.578092 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:05.578325 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "df2edc90c1d545d3be404c3f11954055"
format_stamp: "Formatted at 2026-09-29 01:02:05 on dist-test-slave-7zdl"
server_key: "676a076f07d0bd025bc82ea9781e87d6"
server_key_iv: "2fb2707a5408214928767cdd2e021779"
server_key_version: "encryptionkey@0"
I20260929 01:02:05.578433 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:05.609951 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:05.612941 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:05.613154 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:05.613765 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:05.614594 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:05.614665 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:05.614733 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:05.614782 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:05.624279 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33699
I20260929 01:02:05.624302 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33699 every 8 connection(s)
I20260929 01:02:05.625155 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:02:05.628500 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 1024
I20260929 01:02:05.628607 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:02:05.628940 392 external_mini_cluster.cc:1428] Setting key 4d402d452dfa972871e204835234adfc
Sep 29 01:02:05 dist-test-slave-7zdl krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 01:02:05.640565 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:05.627245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53151 (local address 127.0.98.62:40747)
0929 01:02:05.627514 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:05.627518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:05.628254 (+ 736us) server_negotiation.cc:407] Connection header received
0929 01:02:05.629880 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:05.629883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:05.629926 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:05.630043 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:05.631555 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:05.632132 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:05.633294 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:05.633532 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:05.633666 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:05.636745 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:05.636748 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:05.636750 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:05.636775 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:05.638370 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:05.638968 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:05.638973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:05.638975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:05.639069 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:05.639426 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:05.639429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:05.639431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:05.639616 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:05.639725 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:05.640232 (+ 507us) server_negotiation.cc:299] Negotiation successful
0929 01:02:05.640368 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":104,"threads_started":1}
I20260929 01:02:05.641381 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:05.627678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40747 (local address 127.0.98.1:53151)
0929 01:02:05.628103 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:05.628148 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:05.629681 (+ 1533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:05.630228 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:05.630235 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:05.630662 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:05.631385 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:05.631409 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:05.632255 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:05.632259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:05.633165 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:05.633174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:05.633694 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:05.633698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:05.633901 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:05.634671 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:05.634688 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:05.636563 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:05.638501 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:05.638507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:05.638523 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:05.638854 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:05.639173 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:05.639176 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:05.639178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:05.639324 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:05.639723 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:05.639730 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:05.639986 (+ 256us) client_negotiation.cc:769] Sending connection context
0929 01:02:05.640180 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 01:02:05.640533 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":104,"threads_started":1}
I20260929 01:02:05.642587 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40747
I20260929 01:02:05.642892 1153 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:05.643391 1153 heartbeater.cc:511] Master 127.0.98.62:40747 requested a full tablet report, sending...
I20260929 01:02:05.644923 962 ts_manager.cc:194] Registered new tserver with Master: df2edc90c1d545d3be404c3f11954055 (127.0.98.1:33699)
I20260929 01:02:05.646036 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53151
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 01:02:06.648332 1153 heartbeater.cc:503] Master 127.0.98.62:40747 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260929 01:02:06.776623 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 1.146s user 0.003s sys 0.000s
I20260929 01:02:06.777299 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40747
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:06.897774 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:06.898072 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:06.898154 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:06.902432 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:06.902519 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:06.902616 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:02:06.908133 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40747
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:06.909407 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:06.910473 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:06.920212 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:06.920243 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:06.920243 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:06.920277 1159 server_base.cc:1034] running on GCE node
I20260929 01:02:06.920825 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:06.921476 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:06.922662 1159 hybrid_clock.cc:648] HybridClock initialized: now 1790643726922646 us; error 28 us; skew 500 ppm
Sep 29 01:02:06 dist-test-slave-7zdl krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:06.925330 1159 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:02:06.926159 1159 webserver.cc:460] Webserver started at http://127.0.98.2:40481/ using document root <none> and password file <none>
I20260929 01:02:06.926709 1159 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:06.926797 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:06.927062 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:06.929040 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "9800f5d0ef314166b96cdced9af014dc"
format_stamp: "Formatted at 2026-09-29 01:02:06 on dist-test-slave-7zdl"
server_key: "79fa1d68af49cd14def8cebae4190137"
server_key_iv: "5a79bce449411529f4175b1712b71cb3"
server_key_version: "encryptionkey@0"
I20260929 01:02:06.929584 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9800f5d0ef314166b96cdced9af014dc"
format_stamp: "Formatted at 2026-09-29 01:02:06 on dist-test-slave-7zdl"
server_key: "79fa1d68af49cd14def8cebae4190137"
server_key_iv: "5a79bce449411529f4175b1712b71cb3"
server_key_version: "encryptionkey@0"
I20260929 01:02:06.933116 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 01:02:06.935415 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:06.936604 1159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:02:06.936738 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "9800f5d0ef314166b96cdced9af014dc"
format_stamp: "Formatted at 2026-09-29 01:02:06 on dist-test-slave-7zdl"
server_key: "79fa1d68af49cd14def8cebae4190137"
server_key_iv: "5a79bce449411529f4175b1712b71cb3"
server_key_version: "encryptionkey@0"
I20260929 01:02:06.936838 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:06.946477 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:06.949364 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:06.949553 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:06.950102 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:06.950909 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:06.950979 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:06.951076 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:06.951126 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:06.959770 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34699
I20260929 01:02:06.959798 1287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34699 every 8 connection(s)
I20260929 01:02:06.960606 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:02:06.964051 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 1159
I20260929 01:02:06.964213 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:02:06.964540 392 external_mini_cluster.cc:1428] Setting key 53d037428563e73ef4d2e490ce332b1d
Sep 29 01:02:06 dist-test-slave-7zdl krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 01:02:06.975888 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:06.962407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50665 (local address 127.0.98.62:40747)
0929 01:02:06.962706 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:06.962712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:06.965095 (+ 2383us) server_negotiation.cc:407] Connection header received
0929 01:02:06.966346 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:06.966349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:06.966421 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:06.966570 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:06.967731 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:06.968266 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:06.969413 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:06.969644 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:06.969755 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:06.972247 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:06.972257 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:06.972260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:06.972300 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:06.973844 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:06.974341 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:06.974349 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:06.974351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:06.974424 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:06.974753 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:06.974756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:06.974758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:06.974993 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:06.975101 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:06.975571 (+ 470us) server_negotiation.cc:299] Negotiation successful
0929 01:02:06.975689 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":93,"threads_started":1}
I20260929 01:02:06.976593 1291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:06.962730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40747 (local address 127.0.98.2:50665)
0929 01:02:06.964477 (+ 1747us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:06.964532 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:06.966106 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:06.966734 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:06.966741 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:06.967137 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:06.967587 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:06.967599 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:06.968391 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:06.968396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:06.969287 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:06.969296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:06.969787 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:06.969792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:06.969980 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:06.970588 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:06.970604 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:06.972091 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:06.973964 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:06.973969 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:06.973983 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:06.974230 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:06.974514 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:06.974518 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:06.974520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:06.974651 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:06.975101 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:06.975108 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:06.975333 (+ 225us) client_negotiation.cc:769] Sending connection context
0929 01:02:06.975507 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 01:02:06.975804 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1524,"thread_start_us":98,"threads_started":1}
I20260929 01:02:06.977690 1288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40747
I20260929 01:02:06.977917 1288 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:06.978405 1288 heartbeater.cc:511] Master 127.0.98.62:40747 requested a full tablet report, sending...
I20260929 01:02:06.979466 962 ts_manager.cc:194] Registered new tserver with Master: 9800f5d0ef314166b96cdced9af014dc (127.0.98.2:34699)
I20260929 01:02:06.980419 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260929 01:02:07.536649 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.570s user 0.000s sys 0.003s
I20260929 01:02:07.537168 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40747
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:07.647755 1294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:07.648025 1294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:07.648069 1294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:07.652191 1294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:07.652251 1294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:07.652314 1294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:02:07.657766 1294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40747
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:07.658834 1294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:07.659832 1294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:07.669271 1302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:07.669397 1294 server_base.cc:1034] running on GCE node
W20260929 01:02:07.669275 1299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:07.669348 1300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:07.669976 1294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:07.670591 1294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:07.671746 1294 hybrid_clock.cc:648] HybridClock initialized: now 1790643727671719 us; error 40 us; skew 500 ppm
Sep 29 01:02:07 dist-test-slave-7zdl krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:07.674503 1294 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:02:07.675369 1294 webserver.cc:460] Webserver started at http://127.0.98.3:36971/ using document root <none> and password file <none>
I20260929 01:02:07.675920 1294 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:07.676008 1294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:07.676229 1294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:07.678217 1294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "532cfb65e0f24090b79f3e1ff6472175"
format_stamp: "Formatted at 2026-09-29 01:02:07 on dist-test-slave-7zdl"
server_key: "0a8411720e7f1da13854292754371e28"
server_key_iv: "f8451637e0afa44c902ee8f0cd2931bf"
server_key_version: "encryptionkey@0"
I20260929 01:02:07.678786 1294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "532cfb65e0f24090b79f3e1ff6472175"
format_stamp: "Formatted at 2026-09-29 01:02:07 on dist-test-slave-7zdl"
server_key: "0a8411720e7f1da13854292754371e28"
server_key_iv: "f8451637e0afa44c902ee8f0cd2931bf"
server_key_version: "encryptionkey@0"
I20260929 01:02:07.682404 1294 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:02:07.684862 1309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:07.686069 1294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:07.686196 1294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "532cfb65e0f24090b79f3e1ff6472175"
format_stamp: "Formatted at 2026-09-29 01:02:07 on dist-test-slave-7zdl"
server_key: "0a8411720e7f1da13854292754371e28"
server_key_iv: "f8451637e0afa44c902ee8f0cd2931bf"
server_key_version: "encryptionkey@0"
I20260929 01:02:07.686303 1294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:07.717228 1294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:07.720571 1294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:07.720755 1294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:07.721328 1294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:07.722124 1294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:07.722193 1294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:07.722263 1294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:07.722316 1294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:07.731148 1294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36969
I20260929 01:02:07.731182 1422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36969 every 8 connection(s)
I20260929 01:02:07.732025 1294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:02:07.733418 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 1294
I20260929 01:02:07.733546 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:02:07.733873 392 external_mini_cluster.cc:1428] Setting key 20ae3b582455378b127e030d7e1d3402
Sep 29 01:02:07 dist-test-slave-7zdl krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:07.746302 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:07.733867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34157 (local address 127.0.98.62:40747)
0929 01:02:07.734104 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:07.734109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:07.735172 (+ 1063us) server_negotiation.cc:407] Connection header received
0929 01:02:07.736733 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:07.736737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:07.736792 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:07.736903 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:07.737998 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:07.738541 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:07.739816 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:07.740138 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:07.740251 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:07.742697 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:07.742701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:07.742704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:07.742729 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:07.744324 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:07.744813 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:07.744817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:07.744819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:07.744903 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:07.745248 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:07.745253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:07.745255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:07.745430 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:07.745543 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:07.746004 (+ 461us) server_negotiation.cc:299] Negotiation successful
0929 01:02:07.746121 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":64,"threads_started":1}
I20260929 01:02:07.747237 1426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:07.734397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40747 (local address 127.0.98.3:34157)
0929 01:02:07.735013 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:07.735068 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:07.736537 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:07.737083 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:07.737089 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:07.737421 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:07.737864 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:07.737877 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:07.738662 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:07.738665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:07.739694 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:07.739703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:07.740292 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:07.740296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:07.740472 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:07.741102 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:07.741129 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:07.742545 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:07.744447 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:07.744453 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:07.744466 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:07.744709 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:07.745001 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:07.745004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:07.745006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:07.745153 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:07.745539 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:07.745545 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:07.745774 (+ 229us) client_negotiation.cc:769] Sending connection context
0929 01:02:07.745952 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 01:02:07.746255 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":140,"threads_started":1}
I20260929 01:02:07.748353 1423 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40747
I20260929 01:02:07.748579 1423 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:07.749020 1423 heartbeater.cc:511] Master 127.0.98.62:40747 requested a full tablet report, sending...
I20260929 01:02:07.749974 961 ts_manager.cc:194] Registered new tserver with Master: 532cfb65e0f24090b79f3e1ff6472175 (127.0.98.3:36969)
I20260929 01:02:07.750537 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34157
I20260929 01:02:07.759598 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:02:07.770188 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:07.762828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46242 (local address 127.0.98.62:40747)
0929 01:02:07.762963 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:07.762968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:07.763157 (+ 189us) server_negotiation.cc:407] Connection header received
0929 01:02:07.763275 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:07.763278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:07.763323 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:07.763405 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:07.764107 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:07.764615 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:07.765440 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:07.765667 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:07.765774 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:07.767101 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:07.767105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:07.767108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:07.767139 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:07.768576 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:07.768982 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:07.768986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:07.768988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:07.769064 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:07.769330 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:07.769334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:07.769336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:07.769538 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:07.769680 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:07.769893 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 01:02:07.770018 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 01:02:07.774592 961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:02:07.777446 961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:02:07.792680 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:07.785835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33699 (local address 127.0.0.1:40478)
0929 01:02:07.787280 (+ 1445us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:07.787327 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:07.787459 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:07.788019 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:07.788029 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:07.788059 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:07.788214 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:07.788223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:07.789636 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:07.789639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:07.791050 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:07.791070 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:07.792160 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:07.792163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:07.792267 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:07.792306 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 01:02:07.792415 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 01:02:07.792534 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1270,"thread_start_us":107,"threads_started":1}
I20260929 01:02:07.792716 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:07.786623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40478 (local address 127.0.98.1:33699)
0929 01:02:07.787082 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:07.787093 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:07.787411 (+ 318us) server_negotiation.cc:407] Connection header received
0929 01:02:07.787635 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:07.787643 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:07.787855 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:07.788015 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:07.788344 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:07.789516 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:07.791197 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:07.792021 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:07.792140 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:07.792275 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:07.792456 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 01:02:07.792565 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":95,"threads_started":1}
I20260929 01:02:07.793677 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:07.786698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34699 (local address 127.0.0.1:40728)
0929 01:02:07.788801 (+ 2103us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:07.788814 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:07.788896 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:07.789543 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:07.789549 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:07.789626 (+ 77us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:07.789709 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:07.789750 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:07.791271 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:07.791277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:07.792053 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:07.792062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:07.793276 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:07.793282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:07.793365 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:07.793391 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 01:02:07.793458 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 01:02:07.793524 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2042,"mutex_wait_us":5,"thread_start_us":109,"threads_started":1}
I20260929 01:02:07.794081 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:07.785903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36969 (local address 127.0.0.1:43734)
0929 01:02:07.787937 (+ 2034us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:07.787952 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:07.788034 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:07.788557 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:07.788562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:07.788586 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:07.788702 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:07.788708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:07.791336 (+ 2628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:07.791339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:07.792475 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:07.792484 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:07.793709 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:07.793713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:07.793806 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:07.793826 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 01:02:07.793894 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 01:02:07.793964 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1959,"thread_start_us":85,"threads_started":1}
I20260929 01:02:07.794216 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:07.786365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.0.98.3:36969)
0929 01:02:07.787776 (+ 1411us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:07.787787 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:07.788138 (+ 351us) server_negotiation.cc:407] Connection header received
0929 01:02:07.788212 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:07.788219 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:07.788423 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:07.788574 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:07.788832 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:07.791224 (+ 2392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:07.792620 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:07.793399 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:07.793476 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:07.793609 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:07.793936 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 01:02:07.794043 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1318,"thread_start_us":160,"threads_started":1}
I20260929 01:02:07.794252 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:07.786904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40728 (local address 127.0.98.2:34699)
0929 01:02:07.787274 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:07.787282 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:07.788985 (+ 1703us) server_negotiation.cc:407] Connection header received
0929 01:02:07.789063 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:07.789068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:07.789299 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:07.789427 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:07.789867 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:07.791152 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:07.792213 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:07.792969 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:07.793083 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:07.793213 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:07.794019 (+ 806us) server_negotiation.cc:299] Negotiation successful
0929 01:02:07.794111 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":88,"threads_started":1}
I20260929 01:02:07.795823 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 6627dd651c01441abe29eebde57841d0 (DEFAULT_TABLE table=test-table [id=d9b4edc3dff04d12bf26e1e76df6935f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:07.796846 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6627dd651c01441abe29eebde57841d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:07.796861 1357 tablet_service.cc:1467] Processing CreateTablet for tablet 6627dd651c01441abe29eebde57841d0 (DEFAULT_TABLE table=test-table [id=d9b4edc3dff04d12bf26e1e76df6935f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:07.796861 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 6627dd651c01441abe29eebde57841d0 (DEFAULT_TABLE table=test-table [id=d9b4edc3dff04d12bf26e1e76df6935f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:07.797801 1357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6627dd651c01441abe29eebde57841d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:07.798295 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6627dd651c01441abe29eebde57841d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:07.803450 1442 tablet_bootstrap.cc:492] T 6627dd651c01441abe29eebde57841d0 P df2edc90c1d545d3be404c3f11954055: Bootstrap starting.
I20260929 01:02:07.804662 1443 tablet_bootstrap.cc:492] T 6627dd651c01441abe29eebde57841d0 P 9800f5d0ef314166b96cdced9af014dc: Bootstrap starting.
I20260929 01:02:07.805042 1444 tablet_bootstrap.cc:492] T 6627dd651c01441abe29eebde57841d0 P 532cfb65e0f24090b79f3e1ff6472175: Bootstrap starting.
I20260929 01:02:07.806603 1442 tablet_bootstrap.cc:654] T 6627dd651c01441abe29eebde57841d0 P df2edc90c1d545d3be404c3f11954055: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:07.807484 1442 log.cc:824] T 6627dd651c01441abe29eebde57841d0 P df2edc90c1d545d3be404c3f11954055: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:07.807539 1444 tablet_bootstrap.cc:654] T 6627dd651c01441abe29eebde57841d0 P 532cfb65e0f24090b79f3e1ff6472175: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:07.808835 1444 log.cc:824] T 6627dd651c01441abe29eebde57841d0 P 532cfb65e0f24090b79f3e1ff6472175: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:07.808938 1443 tablet_bootstrap.cc:654] T 6627dd651c01441abe29eebde57841d0 P 9800f5d0ef314166b96cdced9af014dc: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:07.809728 1443 log.cc:824] T 6627dd651c01441abe29eebde57841d0 P 9800f5d0ef314166b96cdced9af014dc: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:07.809829 1442 tablet_bootstrap.cc:492] T 6627dd651c01441abe29eebde57841d0 P df2edc90c1d545d3be404c3f11954055: No bootstrap required, opened a new log
I20260929 01:02:07.810020 1442 ts_tablet_manager.cc:1397] T 6627dd651c01441abe29eebde57841d0 P df2edc90c1d545d3be404c3f11954055: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20260929 01:02:07.810714 1444 tablet_bootstrap.cc:492] T 6627dd651c01441abe29eebde57841d0 P 532cfb65e0f24090b79f3e1ff6472175: No bootstrap required, opened a new log
I20260929 01:02:07.810921 1444 ts_tablet_manager.cc:1397] T 6627dd651c01441abe29eebde57841d0 P 532cfb65e0f24090b79f3e1ff6472175: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260929 01:02:07.811545 1443 tablet_bootstrap.cc:492] T 6627dd651c01441abe29eebde57841d0 P 9800f5d0ef314166b96cdced9af014dc: No bootstrap required, opened a new log
I20260929 01:02:07.811729 1443 ts_tablet_manager.cc:1397] T 6627dd651c01441abe29eebde57841d0 P 9800f5d0ef314166b96cdced9af014dc: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 01:02:07.814458 1442 raft_consensus.cc:348] T 6627dd651c01441abe29eebde57841d0 P df2edc90c1d545d3be404c3f11954055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2edc90c1d545d3be404c3f11954055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33699 } } peers { permanent_uuid: "532cfb65e0f24090b79f3e1ff6472175" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36969 } } peers { permanent_uuid: "9800f5d0ef314166b96cdced9af014dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34699 } }
I20260929 01:02:07.814495 1444 raft_consensus.cc:348] T 6627dd651c01441abe29eebde57841d0 P 532cfb65e0f24090b79f3e1ff6472175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2edc90c1d545d3be404c3f11954055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33699 } } peers { permanent_uuid: "532cfb65e0f24090b79f3e1ff6472175" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36969 } } peers { permanent_uuid: "9800f5d0ef314166b96cdced9af014dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34699 } }
I20260929 01:02:07.814728 1442 raft_consensus.cc:374] T 6627dd651c01441abe29eebde57841d0 P df2edc90c1d545d3be404c3f11954055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:07.814728 1444 raft_consensus.cc:374] T 6627dd651c01441abe29eebde57841d0 P 532cfb65e0f24090b79f3e1ff6472175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:07.814785 1444 raft_consensus.cc:729] T 6627dd651c01441abe29eebde57841d0 P 532cfb65e0f24090b79f3e1ff6472175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 532cfb65e0f24090b79f3e1ff6472175, State: Initialized, Role: FOLLOWER
I20260929 01:02:07.814693 1443 raft_consensus.cc:348] T 6627dd651c01441abe29eebde57841d0 P 9800f5d0ef314166b96cdced9af014dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2edc90c1d545d3be404c3f11954055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33699 } } peers { permanent_uuid: "532cfb65e0f24090b79f3e1ff6472175" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36969 } } peers { permanent_uuid: "9800f5d0ef314166b96cdced9af014dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34699 } }
I20260929 01:02:07.815026 1443 raft_consensus.cc:374] T 6627dd651c01441abe29eebde57841d0 P 9800f5d0ef314166b96cdced9af014dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:07.815122 1443 raft_consensus.cc:729] T 6627dd651c01441abe29eebde57841d0 P 9800f5d0ef314166b96cdced9af014dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9800f5d0ef314166b96cdced9af014dc, State: Initialized, Role: FOLLOWER
I20260929 01:02:07.815217 1442 raft_consensus.cc:729] T 6627dd651c01441abe29eebde57841d0 P df2edc90c1d545d3be404c3f11954055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2edc90c1d545d3be404c3f11954055, State: Initialized, Role: FOLLOWER
I20260929 01:02:07.815327 1444 consensus_queue.cc:260] T 6627dd651c01441abe29eebde57841d0 P 532cfb65e0f24090b79f3e1ff6472175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2edc90c1d545d3be404c3f11954055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33699 } } peers { permanent_uuid: "532cfb65e0f24090b79f3e1ff6472175" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36969 } } peers { permanent_uuid: "9800f5d0ef314166b96cdced9af014dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34699 } }
I20260929 01:02:07.815603 1443 consensus_queue.cc:260] T 6627dd651c01441abe29eebde57841d0 P 9800f5d0ef314166b96cdced9af014dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2edc90c1d545d3be404c3f11954055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33699 } } peers { permanent_uuid: "532cfb65e0f24090b79f3e1ff6472175" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36969 } } peers { permanent_uuid: "9800f5d0ef314166b96cdced9af014dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34699 } }
I20260929 01:02:07.815881 1442 consensus_queue.cc:260] T 6627dd651c01441abe29eebde57841d0 P df2edc90c1d545d3be404c3f11954055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2edc90c1d545d3be404c3f11954055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33699 } } peers { permanent_uuid: "532cfb65e0f24090b79f3e1ff6472175" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36969 } } peers { permanent_uuid: "9800f5d0ef314166b96cdced9af014dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34699 } }
I20260929 01:02:07.816264 1288 heartbeater.cc:503] Master 127.0.98.62:40747 was elected leader, sending a full tablet report...
I20260929 01:02:07.816382 1444 ts_tablet_manager.cc:1428] T 6627dd651c01441abe29eebde57841d0 P 532cfb65e0f24090b79f3e1ff6472175: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260929 01:02:07.816565 1443 ts_tablet_manager.cc:1428] T 6627dd651c01441abe29eebde57841d0 P 9800f5d0ef314166b96cdced9af014dc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 01:02:07.816712 1423 heartbeater.cc:503] Master 127.0.98.62:40747 was elected leader, sending a full tablet report...
I20260929 01:02:07.817883 1442 ts_tablet_manager.cc:1428] T 6627dd651c01441abe29eebde57841d0 P df2edc90c1d545d3be404c3f11954055: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
W20260929 01:02:07.877496 1154 tablet.cc:2369] T 6627dd651c01441abe29eebde57841d0 P df2edc90c1d545d3be404c3f11954055: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:02:07.961949 1289 tablet.cc:2369] T 6627dd651c01441abe29eebde57841d0 P 9800f5d0ef314166b96cdced9af014dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:02:07.983008 1424 tablet.cc:2369] T 6627dd651c01441abe29eebde57841d0 P 532cfb65e0f24090b79f3e1ff6472175: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:02:08.169026 1448 raft_consensus.cc:482] T 6627dd651c01441abe29eebde57841d0 P 532cfb65e0f24090b79f3e1ff6472175 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:02:08.169268 1448 raft_consensus.cc:504] T 6627dd651c01441abe29eebde57841d0 P 532cfb65e0f24090b79f3e1ff6472175 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2edc90c1d545d3be404c3f11954055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33699 } } peers { permanent_uuid: "532cfb65e0f24090b79f3e1ff6472175" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36969 } } peers { permanent_uuid: "9800f5d0ef314166b96cdced9af014dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34699 } }
I20260929 01:02:08.170521 1448 leader_election.cc:288] T 6627dd651c01441abe29eebde57841d0 P 532cfb65e0f24090b79f3e1ff6472175 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df2edc90c1d545d3be404c3f11954055 (127.0.98.1:33699), 9800f5d0ef314166b96cdced9af014dc (127.0.98.2:34699)
I20260929 01:02:08.175383 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:08.170936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34627 (local address 127.0.98.1:33699)
0929 01:02:08.171100 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:08.171106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:08.171383 (+ 277us) server_negotiation.cc:407] Connection header received
0929 01:02:08.171544 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:08.171546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:08.171586 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:08.171644 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:08.171900 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:08.172610 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:08.174110 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:08.174709 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:08.174839 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:08.174900 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:08.175181 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 01:02:08.175253 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 01:02:08.175383 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:08.170932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48553 (local address 127.0.98.2:34699)
0929 01:02:08.171069 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:08.171074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:08.171091 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:02:08.171249 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:08.171252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:08.171298 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:08.171390 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:08.171718 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:08.172491 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:08.174104 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:08.174709 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:08.174839 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:08.174900 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:08.175181 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 01:02:08.175253 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 01:02:08.175405 1426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:08.170822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34699 (local address 127.0.98.3:48553)
0929 01:02:08.170966 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:08.170981 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:08.171099 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:08.171440 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:08.171446 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:08.171463 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:08.171586 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:08.171591 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:08.172593 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:08.172595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:08.173954 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:08.173967 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:08.174848 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:08.174852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:08.175003 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:08.175071 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 01:02:08.175145 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 01:02:08.175246 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":142}
I20260929 01:02:08.175412 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:08.170823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33699 (local address 127.0.98.3:34627)
0929 01:02:08.171294 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:08.171304 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:08.171414 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:08.171679 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:08.171682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:08.171703 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:08.171799 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:08.171803 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:08.172701 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:08.172704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:08.174003 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:08.174013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:08.174848 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:08.174852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:08.175032 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:08.175065 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 01:02:08.175145 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 01:02:08.175246 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":193,"thread_start_us":134,"threads_started":1}
I20260929 01:02:08.176108 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6627dd651c01441abe29eebde57841d0" candidate_uuid: "532cfb65e0f24090b79f3e1ff6472175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9800f5d0ef314166b96cdced9af014dc" is_pre_election: true
I20260929 01:02:08.176147 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6627dd651c01441abe29eebde57841d0" candidate_uuid: "532cfb65e0f24090b79f3e1ff6472175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2edc90c1d545d3be404c3f11954055" is_pre_election: true
I20260929 01:02:08.176403 1242 raft_consensus.cc:2454] T 6627dd651c01441abe29eebde57841d0 P 9800f5d0ef314166b96cdced9af014dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 532cfb65e0f24090b79f3e1ff6472175 in term 0.
I20260929 01:02:08.176422 1107 raft_consensus.cc:2454] T 6627dd651c01441abe29eebde57841d0 P df2edc90c1d545d3be404c3f11954055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 532cfb65e0f24090b79f3e1ff6472175 in term 0.
I20260929 01:02:08.176920 1310 leader_election.cc:302] T 6627dd651c01441abe29eebde57841d0 P 532cfb65e0f24090b79f3e1ff6472175 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 532cfb65e0f24090b79f3e1ff6472175, df2edc90c1d545d3be404c3f11954055; no voters:
I20260929 01:02:08.177234 1448 raft_consensus.cc:2781] T 6627dd651c01441abe29eebde57841d0 P 532cfb65e0f24090b79f3e1ff6472175 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:02:08.177320 1448 raft_consensus.cc:482] T 6627dd651c01441abe29eebde57841d0 P 532cfb65e0f24090b79f3e1ff6472175 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:02:08.177361 1448 raft_consensus.cc:3037] T 6627dd651c01441abe29eebde57841d0 P 532cfb65e0f24090b79f3e1ff6472175 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:08.231762 1448 raft_consensus.cc:504] T 6627dd651c01441abe29eebde57841d0 P 532cfb65e0f24090b79f3e1ff6472175 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2edc90c1d545d3be404c3f11954055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33699 } } peers { permanent_uuid: "532cfb65e0f24090b79f3e1ff6472175" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36969 } } peers { permanent_uuid: "9800f5d0ef314166b96cdced9af014dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34699 } }
I20260929 01:02:08.232585 1448 leader_election.cc:288] T 6627dd651c01441abe29eebde57841d0 P 532cfb65e0f24090b79f3e1ff6472175 [CANDIDATE]: Term 1 election: Requested vote from peers df2edc90c1d545d3be404c3f11954055 (127.0.98.1:33699), 9800f5d0ef314166b96cdced9af014dc (127.0.98.2:34699)
I20260929 01:02:08.232822 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6627dd651c01441abe29eebde57841d0" candidate_uuid: "532cfb65e0f24090b79f3e1ff6472175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df2edc90c1d545d3be404c3f11954055"
I20260929 01:02:08.233027 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6627dd651c01441abe29eebde57841d0" candidate_uuid: "532cfb65e0f24090b79f3e1ff6472175" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9800f5d0ef314166b96cdced9af014dc"
I20260929 01:02:08.233172 1242 raft_consensus.cc:3037] T 6627dd651c01441abe29eebde57841d0 P 9800f5d0ef314166b96cdced9af014dc [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:08.232992 1107 raft_consensus.cc:3037] T 6627dd651c01441abe29eebde57841d0 P df2edc90c1d545d3be404c3f11954055 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:08.235062 1242 raft_consensus.cc:2454] T 6627dd651c01441abe29eebde57841d0 P 9800f5d0ef314166b96cdced9af014dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 532cfb65e0f24090b79f3e1ff6472175 in term 1.
I20260929 01:02:08.235060 1107 raft_consensus.cc:2454] T 6627dd651c01441abe29eebde57841d0 P df2edc90c1d545d3be404c3f11954055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 532cfb65e0f24090b79f3e1ff6472175 in term 1.
I20260929 01:02:08.235455 1311 leader_election.cc:302] T 6627dd651c01441abe29eebde57841d0 P 532cfb65e0f24090b79f3e1ff6472175 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 532cfb65e0f24090b79f3e1ff6472175, 9800f5d0ef314166b96cdced9af014dc; no voters:
I20260929 01:02:08.235649 1448 raft_consensus.cc:2781] T 6627dd651c01441abe29eebde57841d0 P 532cfb65e0f24090b79f3e1ff6472175 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:08.235875 1448 raft_consensus.cc:686] T 6627dd651c01441abe29eebde57841d0 P 532cfb65e0f24090b79f3e1ff6472175 [term 1 LEADER]: Becoming Leader. State: Replica: 532cfb65e0f24090b79f3e1ff6472175, State: Running, Role: LEADER
I20260929 01:02:08.236295 1448 consensus_queue.cc:237] T 6627dd651c01441abe29eebde57841d0 P 532cfb65e0f24090b79f3e1ff6472175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2edc90c1d545d3be404c3f11954055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33699 } } peers { permanent_uuid: "532cfb65e0f24090b79f3e1ff6472175" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36969 } } peers { permanent_uuid: "9800f5d0ef314166b96cdced9af014dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34699 } }
I20260929 01:02:08.240249 961 catalog_manager.cc:5495] T 6627dd651c01441abe29eebde57841d0 P 532cfb65e0f24090b79f3e1ff6472175 reported cstate change: term changed from 0 to 1, leader changed from <none> to 532cfb65e0f24090b79f3e1ff6472175 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "532cfb65e0f24090b79f3e1ff6472175" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df2edc90c1d545d3be404c3f11954055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "532cfb65e0f24090b79f3e1ff6472175" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9800f5d0ef314166b96cdced9af014dc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34699 } health_report { overall_health: UNKNOWN } } }
Sep 29 01:02:08 dist-test-slave-7zdl krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 01:02:08.354072 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:08.344847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40484 (local address 127.0.98.1:33699)
0929 01:02:08.345002 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:08.345007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:08.345153 (+ 146us) server_negotiation.cc:407] Connection header received
0929 01:02:08.345313 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:08.345317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:08.345364 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:08.345459 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:08.346106 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:08.346632 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:08.347503 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:08.347805 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:08.347916 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:08.350311 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:08.350325 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:08.350344 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:08.350375 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:08.351978 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:08.352452 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:08.352459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:08.352461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:08.352551 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:08.353070 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:08.353073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:08.353075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:08.353385 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:08.353612 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:08.353781 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 01:02:08.353902 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 01:02:08 dist-test-slave-7zdl krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643728, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 01:02:08 dist-test-slave-7zdl krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643728, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 01:02:08.416257 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:08.407270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40494 (local address 127.0.98.1:33699)
0929 01:02:08.407407 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:08.407412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:08.407629 (+ 217us) server_negotiation.cc:407] Connection header received
0929 01:02:08.407783 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:08.407789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:08.407857 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:08.407953 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:08.408637 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:08.409529 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:08.410358 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:08.410586 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:08.410670 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:08.413302 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:08.413305 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:08.413307 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:08.413332 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:08.414768 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:08.415196 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:08.415198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:08.415199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:08.415271 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:08.415502 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:08.415504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:08.415505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:08.415662 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:08.415755 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:08.416038 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 01:02:08.416137 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 01:02:08 dist-test-slave-7zdl krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 01:02:08 dist-test-slave-7zdl krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 01:02:08.521093 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:08.512387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40500 (local address 127.0.98.1:33699)
0929 01:02:08.512525 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:08.512530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:08.512701 (+ 171us) server_negotiation.cc:407] Connection header received
0929 01:02:08.512887 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:08.512890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:08.512943 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:08.513031 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:08.513830 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:08.514352 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:08.515329 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:08.515576 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:08.515688 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:08.518235 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:08.518238 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:08.518240 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:08.518264 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:08.519694 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:08.520083 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:08.520085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:08.520086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:08.520154 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:08.520370 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:08.520372 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:08.520373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:08.520533 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:08.520639 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:08.520875 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 01:02:08.520970 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 01:02:08.522436 1067 tablet_service.cc:2908] Rejecting scan request for tablet 6627dd651c01441abe29eebde57841d0: Uninitialized: clean time has not yet been initialized
I20260929 01:02:08.524487 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 1024
I20260929 01:02:08.530479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 1159
I20260929 01:02:08.536262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 1294
I20260929 01:02:08.541867 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 930
2026-09-29T01:02:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (5499 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:02:09 dist-test-slave-7zdl krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:02:09 dist-test-slave-7zdl krb5kdc[1466](info): set up 2 sockets
Sep 29 01:02:09 dist-test-slave-7zdl krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260929 01:02:10.906714 392 mini_kdc.cc:121] Time spent starting KDC: real 2.342s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:02:10 dist-test-slave-7zdl krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:02:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:02:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:02:11.026113 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43845
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:11.133003 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:11.133289 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:11.133363 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:11.137300 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:02:11.137396 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 01:02:11.137429 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:11.137471 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:11.137508 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:02:11.137552 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:02:11.142740 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43845
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:11.143833 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:11.144896 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:11.153438 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:11.153620 1479 server_base.cc:1034] running on GCE node
W20260929 01:02:11.153432 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:11.153432 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:11.154206 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:11.155373 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:11.156533 1479 hybrid_clock.cc:648] HybridClock initialized: now 1790643731156504 us; error 44 us; skew 500 ppm
Sep 29 01:02:11 dist-test-slave-7zdl krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:11.165012 1479 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:02:11.165895 1479 webserver.cc:460] Webserver started at http://127.0.98.62:45265/ using document root <none> and password file <none>
I20260929 01:02:11.166468 1479 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:11.166533 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:11.166762 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:11.168810 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "47695d2b50a2404ebc1671f8e6652f56"
format_stamp: "Formatted at 2026-09-29 01:02:11 on dist-test-slave-7zdl"
server_key: "df0988b216a3c6d00e1461a886c9489a"
server_key_iv: "faec3926086584ee0c998f85d62b3082"
server_key_version: "encryptionkey@0"
I20260929 01:02:11.169435 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "47695d2b50a2404ebc1671f8e6652f56"
format_stamp: "Formatted at 2026-09-29 01:02:11 on dist-test-slave-7zdl"
server_key: "df0988b216a3c6d00e1461a886c9489a"
server_key_iv: "faec3926086584ee0c998f85d62b3082"
server_key_version: "encryptionkey@0"
I20260929 01:02:11.173087 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 01:02:11.175446 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:11.176656 1479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 01:02:11.176797 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "47695d2b50a2404ebc1671f8e6652f56"
format_stamp: "Formatted at 2026-09-29 01:02:11 on dist-test-slave-7zdl"
server_key: "df0988b216a3c6d00e1461a886c9489a"
server_key_iv: "faec3926086584ee0c998f85d62b3082"
server_key_version: "encryptionkey@0"
I20260929 01:02:11.176919 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:11.186712 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:11.189880 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:11.190068 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:11.197821 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43845
I20260929 01:02:11.197871 1556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43845 every 8 connection(s)
I20260929 01:02:11.198740 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:02:11.201444 1557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:11.201733 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 1479
I20260929 01:02:11.201843 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:02:11.202116 392 external_mini_cluster.cc:1428] Setting key f523a2983c89ecfa243e4b82ace362b0
I20260929 01:02:11.207677 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47695d2b50a2404ebc1671f8e6652f56: Bootstrap starting.
I20260929 01:02:11.209862 1557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47695d2b50a2404ebc1671f8e6652f56: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:11.210700 1557 log.cc:824] T 00000000000000000000000000000000 P 47695d2b50a2404ebc1671f8e6652f56: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:02:11 dist-test-slave-7zdl krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:11.212684 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47695d2b50a2404ebc1671f8e6652f56: No bootstrap required, opened a new log
I20260929 01:02:11.215476 1557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47695d2b50a2404ebc1671f8e6652f56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47695d2b50a2404ebc1671f8e6652f56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43845 } }
I20260929 01:02:11.215778 1557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47695d2b50a2404ebc1671f8e6652f56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:11.215864 1557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47695d2b50a2404ebc1671f8e6652f56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47695d2b50a2404ebc1671f8e6652f56, State: Initialized, Role: FOLLOWER
I20260929 01:02:11.216275 1557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47695d2b50a2404ebc1671f8e6652f56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47695d2b50a2404ebc1671f8e6652f56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43845 } }
I20260929 01:02:11.216423 1557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47695d2b50a2404ebc1671f8e6652f56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:02:11.216507 1557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47695d2b50a2404ebc1671f8e6652f56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:02:11.216603 1557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47695d2b50a2404ebc1671f8e6652f56 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:11.216645 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:11.203478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59178 (local address 127.0.98.62:43845)
0929 01:02:11.203909 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:11.203924 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:11.203957 (+ 33us) server_negotiation.cc:407] Connection header received
0929 01:02:11.204678 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:11.204699 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:11.205005 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:11.205346 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:11.206090 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.207097 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.208071 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.208560 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.208735 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:11.211475 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:11.211480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:11.211492 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:11.211516 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:11.213691 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:11.214143 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:11.214153 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:11.214156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:11.214255 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:11.214539 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:11.214543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:11.214545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:11.214979 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:11.215195 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:11.215435 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 01:02:11.215698 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":121,"threads_started":1}
I20260929 01:02:11.217623 1557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47695d2b50a2404ebc1671f8e6652f56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47695d2b50a2404ebc1671f8e6652f56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43845 } }
I20260929 01:02:11.217943 1557 leader_election.cc:302] T 00000000000000000000000000000000 P 47695d2b50a2404ebc1671f8e6652f56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47695d2b50a2404ebc1671f8e6652f56; no voters:
I20260929 01:02:11.218230 1557 leader_election.cc:288] T 00000000000000000000000000000000 P 47695d2b50a2404ebc1671f8e6652f56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:02:11.218325 1562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47695d2b50a2404ebc1671f8e6652f56 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:11.218506 1562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47695d2b50a2404ebc1671f8e6652f56 [term 1 LEADER]: Becoming Leader. State: Replica: 47695d2b50a2404ebc1671f8e6652f56, State: Running, Role: LEADER
I20260929 01:02:11.218863 1562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47695d2b50a2404ebc1671f8e6652f56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47695d2b50a2404ebc1671f8e6652f56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43845 } }
I20260929 01:02:11.219426 1557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47695d2b50a2404ebc1671f8e6652f56 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:02:11.220585 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47695d2b50a2404ebc1671f8e6652f56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47695d2b50a2404ebc1671f8e6652f56. Latest consensus state: current_term: 1 leader_uuid: "47695d2b50a2404ebc1671f8e6652f56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47695d2b50a2404ebc1671f8e6652f56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43845 } } }
I20260929 01:02:11.220628 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47695d2b50a2404ebc1671f8e6652f56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47695d2b50a2404ebc1671f8e6652f56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47695d2b50a2404ebc1671f8e6652f56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43845 } } }
I20260929 01:02:11.220710 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47695d2b50a2404ebc1671f8e6652f56 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:11.220716 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47695d2b50a2404ebc1671f8e6652f56 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:11.221062 1571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:02:11.223667 1571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:02:11.229032 1571 catalog_manager.cc:1347] Generated new cluster ID: ebfa89f5c6c04abe959bf0f918a575e3
I20260929 01:02:11.229110 1571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:02:11.240936 1571 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:02:11.242161 1571 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:02:11.252753 1571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47695d2b50a2404ebc1671f8e6652f56: Generated new TSK 0
I20260929 01:02:11.253526 1571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 01:02:11.281823 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43845
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:11.390673 1583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:11.390967 1583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:11.391016 1583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:11.394860 1583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:11.394905 1583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:11.394942 1583 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:02:11.395010 1583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:02:11.400092 1583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:11.401073 1583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:11.402041 1583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:11.411314 1588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:11.411332 1589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:11.411302 1591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:11.411543 1583 server_base.cc:1034] running on GCE node
I20260929 01:02:11.411970 1583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:11.412513 1583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:11.413663 1583 hybrid_clock.cc:648] HybridClock initialized: now 1790643731413650 us; error 26 us; skew 500 ppm
Sep 29 01:02:11 dist-test-slave-7zdl krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:11.416304 1583 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:02:11.417127 1583 webserver.cc:460] Webserver started at http://127.0.98.1:39737/ using document root <none> and password file <none>
I20260929 01:02:11.417675 1583 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:11.417764 1583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:11.417989 1583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:11.420127 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "8bfba868ebda4a58adbf67edb6b83dc5"
format_stamp: "Formatted at 2026-09-29 01:02:11 on dist-test-slave-7zdl"
server_key: "648be042af7f3dba5fb80b36de79d144"
server_key_iv: "97ccd8e0930e4ff8068fed96449c586b"
server_key_version: "encryptionkey@0"
I20260929 01:02:11.420756 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8bfba868ebda4a58adbf67edb6b83dc5"
format_stamp: "Formatted at 2026-09-29 01:02:11 on dist-test-slave-7zdl"
server_key: "648be042af7f3dba5fb80b36de79d144"
server_key_iv: "97ccd8e0930e4ff8068fed96449c586b"
server_key_version: "encryptionkey@0"
I20260929 01:02:11.424467 1583 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 01:02:11.426728 1598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:11.427918 1583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 01:02:11.428059 1583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "8bfba868ebda4a58adbf67edb6b83dc5"
format_stamp: "Formatted at 2026-09-29 01:02:11 on dist-test-slave-7zdl"
server_key: "648be042af7f3dba5fb80b36de79d144"
server_key_iv: "97ccd8e0930e4ff8068fed96449c586b"
server_key_version: "encryptionkey@0"
I20260929 01:02:11.428164 1583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:11.441145 1583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:11.444265 1583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:11.444455 1583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:11.445797 1583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:11.445868 1583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:11.445940 1583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:11.445986 1583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:02:11 dist-test-slave-7zdl krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:11.456318 1583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35865
I20260929 01:02:11.456380 1713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35865 every 8 connection(s)
I20260929 01:02:11.457321 1583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:02:11.457531 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 1583
I20260929 01:02:11.457653 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:02:11.457980 392 external_mini_cluster.cc:1428] Setting key 4ea1ca68855517907592211cf453fb6e
I20260929 01:02:11.461158 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:11.446757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41345 (local address 127.0.98.62:43845)
0929 01:02:11.446894 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:11.446897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:11.447906 (+ 1009us) server_negotiation.cc:407] Connection header received
0929 01:02:11.448997 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:11.449004 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:11.449075 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:11.449180 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:11.450751 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.451691 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.453387 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.453778 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.453903 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:11.456569 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:11.456573 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:11.456576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:11.456608 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:11.459015 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:11.459578 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:11.459582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:11.459585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:11.459674 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:11.459984 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:11.459990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:11.459994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:11.460246 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:11.460353 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:11.460785 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 01:02:11.460930 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 01:02:11.461992 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:11.447317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43845 (local address 127.0.98.1:41345)
0929 01:02:11.447766 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:11.447806 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:11.448762 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:11.449385 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:11.449396 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:11.449860 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:11.450566 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:11.450585 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.451829 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.451835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:11.453252 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:11.453267 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.453923 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.453926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:11.454066 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:11.454765 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:11.454780 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:11.456383 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:11.459134 (+ 2751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:11.459138 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:11.459149 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:11.459497 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:11.459768 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:11.459770 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:11.459771 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:11.459891 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:11.460349 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:11.460354 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:11.460562 (+ 208us) client_negotiation.cc:769] Sending connection context
0929 01:02:11.460710 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 01:02:11.460938 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":120,"threads_started":1}
I20260929 01:02:11.463320 1714 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43845
I20260929 01:02:11.463461 1714 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:11.464044 1714 heartbeater.cc:511] Master 127.0.98.62:43845 requested a full tablet report, sending...
I20260929 01:02:11.465516 1510 ts_manager.cc:194] Registered new tserver with Master: 8bfba868ebda4a58adbf67edb6b83dc5 (127.0.98.1:35865)
I20260929 01:02:11.466709 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41345
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 01:02:11.475478 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:11.465957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43957 (local address 127.0.98.62:43845)
0929 01:02:11.466060 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:11.466064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:11.468364 (+ 2300us) server_negotiation.cc:407] Connection header received
0929 01:02:11.468513 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:11.468516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:11.468566 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:11.468647 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:11.469100 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.469639 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.470735 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.471035 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.471141 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:11.472411 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:11.472414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:11.472416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:11.472435 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:11.473813 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:11.474314 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:11.474317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:11.474318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:11.474383 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:11.474696 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:11.474698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:11.474699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:11.474895 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:11.474984 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:11.475286 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 01:02:11.475373 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 01:02:11.475672 1721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:11.465901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43845 (local address 127.0.98.1:43957)
0929 01:02:11.468274 (+ 2373us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:11.468286 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:11.468410 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:11.468661 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:11.468664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:11.468888 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:11.468978 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:11.468983 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.469787 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.469792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:11.470621 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:11.470635 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.471154 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.471158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:11.471257 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:11.471790 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:11.471808 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:11.472265 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:11.473942 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:11.473948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:11.473952 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:11.474221 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:11.474483 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:11.474489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:11.474494 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:11.474606 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:11.474990 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:11.474995 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:11.475107 (+ 112us) client_negotiation.cc:769] Sending connection context
0929 01:02:11.475238 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 01:02:11.475406 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2302,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 01:02:11.486626 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43845
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:11.594839 1723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:11.595113 1723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:11.595158 1723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:11.599068 1723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:11.599123 1723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:11.599160 1723 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:02:11.599192 1723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:02:11.604209 1723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:11.605141 1723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:11.606065 1723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:11.615388 1729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:11.615448 1731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:11.615415 1728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:11.615589 1723 server_base.cc:1034] running on GCE node
I20260929 01:02:11.616070 1723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:11.616743 1723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:11.617915 1723 hybrid_clock.cc:648] HybridClock initialized: now 1790643731617901 us; error 29 us; skew 500 ppm
Sep 29 01:02:11 dist-test-slave-7zdl krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:11.620779 1723 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:02:11.621635 1723 webserver.cc:460] Webserver started at http://127.0.98.2:46819/ using document root <none> and password file <none>
I20260929 01:02:11.622189 1723 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:11.622254 1723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:11.622503 1723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:11.624570 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "abd6cb9a241b4f44942ee39b5108cd3f"
format_stamp: "Formatted at 2026-09-29 01:02:11 on dist-test-slave-7zdl"
server_key: "c681d508b6a87bb82dff9a13baad8305"
server_key_iv: "dea5b4fdf7410e1cade20d45db2a53b3"
server_key_version: "encryptionkey@0"
I20260929 01:02:11.625129 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "abd6cb9a241b4f44942ee39b5108cd3f"
format_stamp: "Formatted at 2026-09-29 01:02:11 on dist-test-slave-7zdl"
server_key: "c681d508b6a87bb82dff9a13baad8305"
server_key_iv: "dea5b4fdf7410e1cade20d45db2a53b3"
server_key_version: "encryptionkey@0"
I20260929 01:02:11.628731 1723 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:02:11.631208 1738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:11.632503 1723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:02:11.632622 1723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "abd6cb9a241b4f44942ee39b5108cd3f"
format_stamp: "Formatted at 2026-09-29 01:02:11 on dist-test-slave-7zdl"
server_key: "c681d508b6a87bb82dff9a13baad8305"
server_key_iv: "dea5b4fdf7410e1cade20d45db2a53b3"
server_key_version: "encryptionkey@0"
I20260929 01:02:11.632726 1723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:11.689530 1723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:11.693066 1723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:11.693267 1723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:11.694705 1723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:11.694752 1723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:11.694840 1723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:11.694876 1723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:02:11 dist-test-slave-7zdl krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:11.705718 1723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40427
I20260929 01:02:11.706967 1723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:02:11.709542 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:11.695604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45401 (local address 127.0.98.62:43845)
0929 01:02:11.695748 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:11.695753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:11.696611 (+ 858us) server_negotiation.cc:407] Connection header received
0929 01:02:11.697674 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:11.697680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:11.697745 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:11.697811 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:11.699187 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.699809 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.701061 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.701283 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.701396 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:11.704618 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:11.704624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:11.704630 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:11.704662 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:11.706549 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:11.707189 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:11.707221 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:11.707225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:11.707351 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:11.708144 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:11.708149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:11.708165 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:11.708436 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:11.708549 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:11.709208 (+ 659us) server_negotiation.cc:299] Negotiation successful
0929 01:02:11.709408 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:02:11.710420 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:11.696023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43845 (local address 127.0.98.2:45401)
0929 01:02:11.696442 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:11.696490 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:11.697427 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:11.698200 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:11.698207 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:11.698508 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:11.699038 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:11.699053 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.699953 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.699956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:11.700934 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:11.700943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.701440 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.701444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:11.701605 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:11.702170 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:11.702185 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:11.704404 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:11.706678 (+ 2274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:11.706684 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:11.706697 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:11.707089 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:11.707516 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:11.707520 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:11.707522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:11.707792 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:11.708571 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:11.708579 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:11.708910 (+ 331us) client_negotiation.cc:769] Sending connection context
0929 01:02:11.709110 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 01:02:11.709443 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":118,"threads_started":1}
I20260929 01:02:11.707762 1853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40427 every 8 connection(s)
I20260929 01:02:11.711869 1854 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43845
I20260929 01:02:11.712023 1854 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:11.712568 1854 heartbeater.cc:511] Master 127.0.98.62:43845 requested a full tablet report, sending...
I20260929 01:02:11.713140 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 1723
I20260929 01:02:11.713251 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:02:11.713557 392 external_mini_cluster.cc:1428] Setting key ecabff229c82519207d5b0399087a92f
I20260929 01:02:11.713637 1511 ts_manager.cc:194] Registered new tserver with Master: abd6cb9a241b4f44942ee39b5108cd3f (127.0.98.2:40427)
I20260929 01:02:11.714388 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45401
I20260929 01:02:11.722368 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:11.714682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59051 (local address 127.0.98.62:43845)
0929 01:02:11.714778 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:11.714780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:11.714790 (+ 10us) server_negotiation.cc:407] Connection header received
0929 01:02:11.714863 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:11.714866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:11.714903 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:11.714998 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:11.715432 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.716191 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.717212 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.717537 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.717622 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:11.718940 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:11.718943 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:11.718945 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:11.718964 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:11.720648 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:11.721131 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:11.721134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:11.721135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:11.721230 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:11.721595 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:11.721597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:11.721598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:11.721775 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:11.721850 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:11.722176 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 01:02:11.722263 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 01:02:11.722492 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:11.714173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43845 (local address 127.0.98.2:59051)
0929 01:02:11.714436 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:11.714449 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:11.714533 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:11.715015 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:11.715018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:11.715215 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:11.715327 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:11.715333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.716338 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.716342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:11.717095 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:11.717103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.717654 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.717659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:11.717779 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:11.718240 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:11.718260 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:11.718764 (+ 504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:11.720760 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:11.720765 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:11.720769 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:11.721037 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:11.721331 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:11.721336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:11.721341 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:11.721448 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:11.721874 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:11.721878 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:11.721984 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 01:02:11.722100 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 01:02:11.722243 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 01:02:11.741291 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43845
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:11.848574 1863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:11.848812 1863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:11.848886 1863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:11.854051 1863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:11.854130 1863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:11.854207 1863 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:02:11.854290 1863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:02:11.861737 1863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:11.862679 1863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:11.863739 1863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:11.873036 1871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:11.873062 1868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:11.873267 1863 server_base.cc:1034] running on GCE node
W20260929 01:02:11.873070 1869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:11.873821 1863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:11.874462 1863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:11.875641 1863 hybrid_clock.cc:648] HybridClock initialized: now 1790643731875628 us; error 25 us; skew 500 ppm
Sep 29 01:02:11 dist-test-slave-7zdl krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:11.878160 1863 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:02:11.879015 1863 webserver.cc:460] Webserver started at http://127.0.98.3:37439/ using document root <none> and password file <none>
I20260929 01:02:11.879557 1863 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:11.879647 1863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:11.879890 1863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:11.881946 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "411cd7da2c6248b3991af98fa63c46fe"
format_stamp: "Formatted at 2026-09-29 01:02:11 on dist-test-slave-7zdl"
server_key: "c3371119819d37aa3c4570405121258a"
server_key_iv: "ab55a5e1d5f5aed542b8c874f7830b97"
server_key_version: "encryptionkey@0"
I20260929 01:02:11.882490 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "411cd7da2c6248b3991af98fa63c46fe"
format_stamp: "Formatted at 2026-09-29 01:02:11 on dist-test-slave-7zdl"
server_key: "c3371119819d37aa3c4570405121258a"
server_key_iv: "ab55a5e1d5f5aed542b8c874f7830b97"
server_key_version: "encryptionkey@0"
I20260929 01:02:11.886111 1863 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 01:02:11.888523 1878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:11.889681 1863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:02:11.889797 1863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "411cd7da2c6248b3991af98fa63c46fe"
format_stamp: "Formatted at 2026-09-29 01:02:11 on dist-test-slave-7zdl"
server_key: "c3371119819d37aa3c4570405121258a"
server_key_iv: "ab55a5e1d5f5aed542b8c874f7830b97"
server_key_version: "encryptionkey@0"
I20260929 01:02:11.889899 1863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:11.902199 1863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:11.904896 1863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:11.905092 1863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:11.906486 1863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:11.906530 1863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:11.906612 1863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:11.906661 1863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:02:11 dist-test-slave-7zdl krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:11.917457 1863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45107
I20260929 01:02:11.917770 1993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45107 every 8 connection(s)
I20260929 01:02:11.918336 1863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:02:11.920253 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:11.907345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53491 (local address 127.0.98.62:43845)
0929 01:02:11.907469 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:11.907473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:11.908453 (+ 980us) server_negotiation.cc:407] Connection header received
0929 01:02:11.909481 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:11.909485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:11.909543 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:11.909606 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:11.910899 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.911657 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.913583 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.913953 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.914076 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:11.916535 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:11.916539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:11.916541 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:11.916570 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:11.918239 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:11.918818 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:11.918825 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:11.918830 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:11.918924 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:11.919302 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:11.919305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:11.919306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:11.919525 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:11.919634 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:11.920047 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 01:02:11.920141 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 01:02:11.921060 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:11.907857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43845 (local address 127.0.98.3:53491)
0929 01:02:11.908308 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:11.908347 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:11.909237 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:11.909925 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:11.909931 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:11.910296 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:11.910735 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:11.910747 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.911822 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.911830 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:11.913448 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:11.913463 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.914114 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.914118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:11.914310 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:11.914968 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:11.914985 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:11.916358 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:11.918355 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:11.918361 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:11.918375 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:11.918689 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:11.919041 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:11.919044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:11.919046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:11.919195 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:11.919651 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:11.919655 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:11.919852 (+ 197us) client_negotiation.cc:769] Sending connection context
0929 01:02:11.920011 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 01:02:11.920233 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":131,"threads_started":1}
I20260929 01:02:11.922219 1994 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43845
I20260929 01:02:11.922349 1994 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:11.922950 1994 heartbeater.cc:511] Master 127.0.98.62:43845 requested a full tablet report, sending...
I20260929 01:02:11.924012 1510 ts_manager.cc:194] Registered new tserver with Master: 411cd7da2c6248b3991af98fa63c46fe (127.0.98.3:45107)
I20260929 01:02:11.924780 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53491
I20260929 01:02:11.926734 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 1863
I20260929 01:02:11.926873 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:02:11.927126 392 external_mini_cluster.cc:1428] Setting key e91d3b33abb71d80166f5a6a7b0b0fa0
I20260929 01:02:11.929169 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:02:11.931876 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:11.924490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43845 (local address 127.0.98.3:33829)
0929 01:02:11.924766 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:11.924780 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:11.924877 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:11.925307 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:11.925310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:11.925460 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:11.925557 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:11.925562 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.926410 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.926413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:11.927056 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:11.927065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.927526 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.927529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:11.927625 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:11.928149 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:11.928163 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:11.928616 (+ 453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:11.930283 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:11.930289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:11.930294 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:11.930549 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:11.930831 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:11.930836 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:11.930841 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:11.930942 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:11.931465 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:11.931488 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:11.931560 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 01:02:11.931650 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 01:02:11.931749 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":84,"threads_started":1}
I20260929 01:02:11.932057 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:11.924569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33829 (local address 127.0.98.62:43845)
0929 01:02:11.924678 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:11.924681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:11.924885 (+ 204us) server_negotiation.cc:407] Connection header received
0929 01:02:11.924959 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:11.924961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:11.925016 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:11.925096 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:11.925668 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.926157 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.927165 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.927402 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.927515 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:11.928739 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:11.928743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:11.928745 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:11.928768 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:11.930179 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:11.930637 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:11.930639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:11.930640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:11.930704 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:11.931052 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:11.931056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:11.931061 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:11.931281 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:11.931407 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:11.931731 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 01:02:11.931853 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 29 01:02:11 dist-test-slave-7zdl krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643731, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 01:02:11 dist-test-slave-7zdl krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643731, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:11.954427 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:11.946757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34966 (local address 127.0.98.62:43845)
0929 01:02:11.946923 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:11.946926 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:11.946936 (+ 10us) server_negotiation.cc:407] Connection header received
0929 01:02:11.946968 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:11.946970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:11.947002 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:11.947089 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:11.947764 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.948410 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.949276 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.949488 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.949594 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:11.951931 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:11.951934 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:11.951936 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:11.951956 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:11.953162 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:11.953509 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:11.953511 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:11.953512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:11.953578 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:11.953797 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:11.953799 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:11.953800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:11.953957 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:11.954075 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:11.954243 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 01:02:11.954322 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260929 01:02:11.956955 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:02:11.960083 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:02:11.973901 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:11.968060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40427 (local address 127.0.0.1:37400)
0929 01:02:11.968653 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:11.968702 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:11.969189 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:11.969814 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:11.969819 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:11.969840 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:11.969959 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:11.969973 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.971091 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.971097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:11.972473 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:11.972483 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.973362 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.973366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:11.973462 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:11.973495 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 01:02:11.973634 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 01:02:11.973740 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":64,"thread_start_us":98,"threads_started":1}
I20260929 01:02:11.974090 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:11.968930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36160 (local address 127.0.98.1:35865)
0929 01:02:11.969407 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:11.969413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:11.969657 (+ 244us) server_negotiation.cc:407] Connection header received
0929 01:02:11.969747 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:11.969753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:11.969912 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:11.970061 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:11.970257 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.971372 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.972740 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.973529 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.973641 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:11.973764 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:11.973848 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 01:02:11.973929 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":69,"threads_started":1}
I20260929 01:02:11.974203 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:11.968249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37400 (local address 127.0.98.2:40427)
0929 01:02:11.969264 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:11.969278 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:11.969302 (+ 24us) server_negotiation.cc:407] Connection header received
0929 01:02:11.969376 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:11.969382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:11.969591 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:11.969799 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:11.970114 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.970983 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.972602 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.973246 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.973364 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:11.973522 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:11.973964 (+ 442us) server_negotiation.cc:299] Negotiation successful
0929 01:02:11.974078 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":906,"thread_start_us":96,"threads_started":1}
I20260929 01:02:11.974222 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:11.968477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45107 (local address 127.0.0.1:47048)
0929 01:02:11.968757 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:11.968771 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:11.968888 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:11.969956 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:11.969963 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:11.969982 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:11.970087 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:11.970093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.971475 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.971479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:11.972754 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:11.972762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.973840 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.973843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:11.973926 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:11.973946 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 01:02:11.974014 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 01:02:11.974071 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":132,"thread_start_us":115,"threads_started":1}
I20260929 01:02:11.974653 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:11.968805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35865 (local address 127.0.0.1:36160)
0929 01:02:11.969537 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:11.969562 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:11.969652 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:11.970058 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:11.970061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:11.970082 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:11.970170 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:11.970175 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.971523 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.971526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:11.972627 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:11.972636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.973654 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.973657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:11.973734 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:11.973755 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 01:02:11.973825 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 01:02:11.973873 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":76,"thread_start_us":89,"threads_started":1}
I20260929 01:02:11.974732 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:11.968909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47048 (local address 127.0.98.3:45107)
0929 01:02:11.969381 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:11.969391 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:11.969413 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:02:11.969490 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:11.969496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:11.969698 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:11.969846 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:11.970209 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.971360 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.972863 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:11.973554 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:11.973641 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:11.973781 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:11.974491 (+ 710us) server_negotiation.cc:299] Negotiation successful
0929 01:02:11.974604 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":94,"threads_started":1}
I20260929 01:02:11.977098 1928 tablet_service.cc:1467] Processing CreateTablet for tablet 2a19bcf5701d4671be641aed6f26f835 (DEFAULT_TABLE table=test-table [id=e9c9b13eb36e491a88456ea869dcb3a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:11.977394 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 2a19bcf5701d4671be641aed6f26f835 (DEFAULT_TABLE table=test-table [id=e9c9b13eb36e491a88456ea869dcb3a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:11.977404 1788 tablet_service.cc:1467] Processing CreateTablet for tablet 2a19bcf5701d4671be641aed6f26f835 (DEFAULT_TABLE table=test-table [id=e9c9b13eb36e491a88456ea869dcb3a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:11.978061 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a19bcf5701d4671be641aed6f26f835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:11.978348 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a19bcf5701d4671be641aed6f26f835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:11.978348 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a19bcf5701d4671be641aed6f26f835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:11.984743 2017 tablet_bootstrap.cc:492] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe: Bootstrap starting.
I20260929 01:02:11.986218 2019 tablet_bootstrap.cc:492] T 2a19bcf5701d4671be641aed6f26f835 P abd6cb9a241b4f44942ee39b5108cd3f: Bootstrap starting.
I20260929 01:02:11.986894 2017 tablet_bootstrap.cc:654] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:11.987130 2018 tablet_bootstrap.cc:492] T 2a19bcf5701d4671be641aed6f26f835 P 8bfba868ebda4a58adbf67edb6b83dc5: Bootstrap starting.
I20260929 01:02:11.987681 2017 log.cc:824] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:11.988176 2019 tablet_bootstrap.cc:654] T 2a19bcf5701d4671be641aed6f26f835 P abd6cb9a241b4f44942ee39b5108cd3f: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:11.988965 2019 log.cc:824] T 2a19bcf5701d4671be641aed6f26f835 P abd6cb9a241b4f44942ee39b5108cd3f: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:11.989015 2018 tablet_bootstrap.cc:654] T 2a19bcf5701d4671be641aed6f26f835 P 8bfba868ebda4a58adbf67edb6b83dc5: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:11.989620 2017 tablet_bootstrap.cc:492] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe: No bootstrap required, opened a new log
I20260929 01:02:11.989796 2018 log.cc:824] T 2a19bcf5701d4671be641aed6f26f835 P 8bfba868ebda4a58adbf67edb6b83dc5: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:11.989827 2017 ts_tablet_manager.cc:1397] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260929 01:02:11.991032 2019 tablet_bootstrap.cc:492] T 2a19bcf5701d4671be641aed6f26f835 P abd6cb9a241b4f44942ee39b5108cd3f: No bootstrap required, opened a new log
I20260929 01:02:11.991209 2019 ts_tablet_manager.cc:1397] T 2a19bcf5701d4671be641aed6f26f835 P abd6cb9a241b4f44942ee39b5108cd3f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 01:02:11.991484 2018 tablet_bootstrap.cc:492] T 2a19bcf5701d4671be641aed6f26f835 P 8bfba868ebda4a58adbf67edb6b83dc5: No bootstrap required, opened a new log
I20260929 01:02:11.991654 2018 ts_tablet_manager.cc:1397] T 2a19bcf5701d4671be641aed6f26f835 P 8bfba868ebda4a58adbf67edb6b83dc5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 01:02:11.993718 2017 raft_consensus.cc:348] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:11.993968 2017 raft_consensus.cc:374] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:11.994052 2017 raft_consensus.cc:729] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 411cd7da2c6248b3991af98fa63c46fe, State: Initialized, Role: FOLLOWER
I20260929 01:02:11.994494 2017 consensus_queue.cc:260] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:11.994997 2019 raft_consensus.cc:348] T 2a19bcf5701d4671be641aed6f26f835 P abd6cb9a241b4f44942ee39b5108cd3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:11.995062 2018 raft_consensus.cc:348] T 2a19bcf5701d4671be641aed6f26f835 P 8bfba868ebda4a58adbf67edb6b83dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:11.995227 2019 raft_consensus.cc:374] T 2a19bcf5701d4671be641aed6f26f835 P abd6cb9a241b4f44942ee39b5108cd3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:11.995273 2018 raft_consensus.cc:374] T 2a19bcf5701d4671be641aed6f26f835 P 8bfba868ebda4a58adbf67edb6b83dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:11.995304 2019 raft_consensus.cc:729] T 2a19bcf5701d4671be641aed6f26f835 P abd6cb9a241b4f44942ee39b5108cd3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abd6cb9a241b4f44942ee39b5108cd3f, State: Initialized, Role: FOLLOWER
I20260929 01:02:11.995334 2018 raft_consensus.cc:729] T 2a19bcf5701d4671be641aed6f26f835 P 8bfba868ebda4a58adbf67edb6b83dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bfba868ebda4a58adbf67edb6b83dc5, State: Initialized, Role: FOLLOWER
I20260929 01:02:11.995461 2017 ts_tablet_manager.cc:1428] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260929 01:02:11.995869 1994 heartbeater.cc:503] Master 127.0.98.62:43845 was elected leader, sending a full tablet report...
I20260929 01:02:11.995831 2018 consensus_queue.cc:260] T 2a19bcf5701d4671be641aed6f26f835 P 8bfba868ebda4a58adbf67edb6b83dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:11.996028 2019 consensus_queue.cc:260] T 2a19bcf5701d4671be641aed6f26f835 P abd6cb9a241b4f44942ee39b5108cd3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:11.996683 2018 ts_tablet_manager.cc:1428] T 2a19bcf5701d4671be641aed6f26f835 P 8bfba868ebda4a58adbf67edb6b83dc5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 01:02:11.996708 1854 heartbeater.cc:503] Master 127.0.98.62:43845 was elected leader, sending a full tablet report...
I20260929 01:02:11.996891 2019 ts_tablet_manager.cc:1428] T 2a19bcf5701d4671be641aed6f26f835 P abd6cb9a241b4f44942ee39b5108cd3f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 01:02:11.996963 1714 heartbeater.cc:503] Master 127.0.98.62:43845 was elected leader, sending a full tablet report...
I20260929 01:02:12.154032 2023 raft_consensus.cc:482] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:02:12.154259 2023 raft_consensus.cc:504] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:12.155481 2023 leader_election.cc:288] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abd6cb9a241b4f44942ee39b5108cd3f (127.0.98.2:40427), 8bfba868ebda4a58adbf67edb6b83dc5 (127.0.98.1:35865)
I20260929 01:02:12.159111 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:12.155803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49171 (local address 127.0.98.2:40427)
0929 01:02:12.155945 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:12.155952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:12.155969 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:02:12.156183 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:12.156190 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:12.156251 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:12.156332 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:12.156620 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.157139 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.158163 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.158564 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.158684 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:12.158750 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:12.158925 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 01:02:12.158994 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 01:02:12.159111 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:12.155728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40427 (local address 127.0.98.3:49171)
0929 01:02:12.155904 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:12.155921 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:12.156045 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:12.156369 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:12.156372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:12.156387 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:12.156507 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:12.156512 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.157250 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.157256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:12.158039 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:12.158049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.158682 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.158686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:12.158781 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:12.158830 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 01:02:12.158906 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 01:02:12.158980 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 01:02:12.159755 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a19bcf5701d4671be641aed6f26f835" candidate_uuid: "411cd7da2c6248b3991af98fa63c46fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" is_pre_election: true
I20260929 01:02:12.160071 1808 raft_consensus.cc:2454] T 2a19bcf5701d4671be641aed6f26f835 P abd6cb9a241b4f44942ee39b5108cd3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 411cd7da2c6248b3991af98fa63c46fe in term 0.
I20260929 01:02:12.160143 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:12.156024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35865 (local address 127.0.98.3:55727)
0929 01:02:12.156481 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:12.156492 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:12.156607 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:12.156837 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:12.156840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:12.156855 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:12.156991 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:12.156996 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.157691 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.157698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:12.158878 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:12.158888 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.159732 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.159736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:12.159826 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:12.159845 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 01:02:12.159939 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 01:02:12.160003 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":170,"threads_started":1}
I20260929 01:02:12.160128 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:12.156183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55727 (local address 127.0.98.1:35865)
0929 01:02:12.156317 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:12.156322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:12.156579 (+ 257us) server_negotiation.cc:407] Connection header received
0929 01:02:12.156697 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:12.156700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:12.156748 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:12.156829 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:12.157103 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.157585 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.159009 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.159597 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.159726 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:12.159783 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:12.159953 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 01:02:12.160022 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 01:02:12.160581 1879 leader_election.cc:302] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 411cd7da2c6248b3991af98fa63c46fe, abd6cb9a241b4f44942ee39b5108cd3f; no voters:
I20260929 01:02:12.160843 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a19bcf5701d4671be641aed6f26f835" candidate_uuid: "411cd7da2c6248b3991af98fa63c46fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" is_pre_election: true
I20260929 01:02:12.160890 2023 raft_consensus.cc:2781] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:02:12.160979 2023 raft_consensus.cc:482] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:02:12.161023 2023 raft_consensus.cc:3037] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:12.161160 1668 raft_consensus.cc:2454] T 2a19bcf5701d4671be641aed6f26f835 P 8bfba868ebda4a58adbf67edb6b83dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 411cd7da2c6248b3991af98fa63c46fe in term 0.
I20260929 01:02:12.162118 2023 raft_consensus.cc:504] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:12.162395 2023 leader_election.cc:288] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe [CANDIDATE]: Term 1 election: Requested vote from peers abd6cb9a241b4f44942ee39b5108cd3f (127.0.98.2:40427), 8bfba868ebda4a58adbf67edb6b83dc5 (127.0.98.1:35865)
I20260929 01:02:12.162727 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a19bcf5701d4671be641aed6f26f835" candidate_uuid: "411cd7da2c6248b3991af98fa63c46fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abd6cb9a241b4f44942ee39b5108cd3f"
I20260929 01:02:12.162789 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a19bcf5701d4671be641aed6f26f835" candidate_uuid: "411cd7da2c6248b3991af98fa63c46fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bfba868ebda4a58adbf67edb6b83dc5"
I20260929 01:02:12.162899 1808 raft_consensus.cc:3037] T 2a19bcf5701d4671be641aed6f26f835 P abd6cb9a241b4f44942ee39b5108cd3f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:12.163036 1668 raft_consensus.cc:3037] T 2a19bcf5701d4671be641aed6f26f835 P 8bfba868ebda4a58adbf67edb6b83dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:12.164142 1808 raft_consensus.cc:2454] T 2a19bcf5701d4671be641aed6f26f835 P abd6cb9a241b4f44942ee39b5108cd3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 411cd7da2c6248b3991af98fa63c46fe in term 1.
I20260929 01:02:12.164207 1668 raft_consensus.cc:2454] T 2a19bcf5701d4671be641aed6f26f835 P 8bfba868ebda4a58adbf67edb6b83dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 411cd7da2c6248b3991af98fa63c46fe in term 1.
I20260929 01:02:12.164474 1879 leader_election.cc:302] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 411cd7da2c6248b3991af98fa63c46fe, abd6cb9a241b4f44942ee39b5108cd3f; no voters:
I20260929 01:02:12.164640 2023 raft_consensus.cc:2781] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:12.164851 2023 raft_consensus.cc:686] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe [term 1 LEADER]: Becoming Leader. State: Replica: 411cd7da2c6248b3991af98fa63c46fe, State: Running, Role: LEADER
I20260929 01:02:12.165185 2023 consensus_queue.cc:237] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:12.168241 1510 catalog_manager.cc:5495] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 411cd7da2c6248b3991af98fa63c46fe (127.0.98.3). New cstate: current_term: 1 leader_uuid: "411cd7da2c6248b3991af98fa63c46fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } health_report { overall_health: HEALTHY } } }
W20260929 01:02:12.169307 1995 tablet.cc:2369] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:02:12.208848 1715 tablet.cc:2369] T 2a19bcf5701d4671be641aed6f26f835 P 8bfba868ebda4a58adbf67edb6b83dc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:02:12.209668 1855 tablet.cc:2369] T 2a19bcf5701d4671be641aed6f26f835 P abd6cb9a241b4f44942ee39b5108cd3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:02:12.211901 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:12.207430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47062 (local address 127.0.98.3:45107)
0929 01:02:12.207559 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:12.207563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:12.207576 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:02:12.207777 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:12.207779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:12.207832 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:12.207915 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:02:12.208200 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.209017 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.210428 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.210649 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.210768 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:12.211100 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:02:12.211505 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:02:12.211624 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:12.211696 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 01:02:12.211750 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 01:02:12.217216 1808 raft_consensus.cc:1261] T 2a19bcf5701d4671be641aed6f26f835 P abd6cb9a241b4f44942ee39b5108cd3f [term 1 FOLLOWER]: Refusing update from remote peer 411cd7da2c6248b3991af98fa63c46fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:12.217216 1668 raft_consensus.cc:1261] T 2a19bcf5701d4671be641aed6f26f835 P 8bfba868ebda4a58adbf67edb6b83dc5 [term 1 FOLLOWER]: Refusing update from remote peer 411cd7da2c6248b3991af98fa63c46fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:12.217877 2027 consensus_queue.cc:1035] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:12.218077 2023 consensus_queue.cc:1035] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:12.226127 2031 mvcc.cc:205] Tried to move back new op lower bound from 7334476727155109888 to 7334476726951366656. Current Snapshot: MvccSnapshot[applied={T|T < 7334476727155109888}]
I20260929 01:02:12.237780 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:12.237960 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:12.240391 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47695d2b50a2404ebc1671f8e6652f56: Sending DeleteTablet for 3 replicas of tablet 2a19bcf5701d4671be641aed6f26f835
I20260929 01:02:12.241173 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a19bcf5701d4671be641aed6f26f835 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36160
I20260929 01:02:12.241168 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a19bcf5701d4671be641aed6f26f835 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47048
I20260929 01:02:12.241636 2041 tablet_replica.cc:330] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe: stopping tablet replica
I20260929 01:02:12.241627 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a19bcf5701d4671be641aed6f26f835 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37400
I20260929 01:02:12.241926 2041 raft_consensus.cc:2229] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:02:12.242082 2042 tablet_replica.cc:330] T 2a19bcf5701d4671be641aed6f26f835 P abd6cb9a241b4f44942ee39b5108cd3f: stopping tablet replica
I20260929 01:02:12.242306 2041 raft_consensus.cc:2258] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:12.242484 2042 raft_consensus.cc:2229] T 2a19bcf5701d4671be641aed6f26f835 P abd6cb9a241b4f44942ee39b5108cd3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:12.242708 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 01:02:12.242815 2042 raft_consensus.cc:2258] T 2a19bcf5701d4671be641aed6f26f835 P abd6cb9a241b4f44942ee39b5108cd3f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 01:02:12.243145 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:02:12.243623 2041 ts_tablet_manager.cc:1905] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:12.245743 2042 ts_tablet_manager.cc:1905] T 2a19bcf5701d4671be641aed6f26f835 P abd6cb9a241b4f44942ee39b5108cd3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:12.246435 2041 ts_tablet_manager.cc:1918] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:02:12.246526 2041 log.cc:1196] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/2a19bcf5701d4671be641aed6f26f835
I20260929 01:02:12.246858 2041 ts_tablet_manager.cc:1939] T 2a19bcf5701d4671be641aed6f26f835 P 411cd7da2c6248b3991af98fa63c46fe: Deleting consensus metadata
I20260929 01:02:12.247884 2040 tablet_replica.cc:330] T 2a19bcf5701d4671be641aed6f26f835 P 8bfba868ebda4a58adbf67edb6b83dc5: stopping tablet replica
I20260929 01:02:12.247982 1497 catalog_manager.cc:4841] TS 411cd7da2c6248b3991af98fa63c46fe (127.0.98.3:45107): tablet 2a19bcf5701d4671be641aed6f26f835 (table test-table [id=e9c9b13eb36e491a88456ea869dcb3a7]) successfully deleted
I20260929 01:02:12.248170 2040 raft_consensus.cc:2229] T 2a19bcf5701d4671be641aed6f26f835 P 8bfba868ebda4a58adbf67edb6b83dc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:12.248409 2040 raft_consensus.cc:2258] T 2a19bcf5701d4671be641aed6f26f835 P 8bfba868ebda4a58adbf67edb6b83dc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:12.248413 1788 tablet_service.cc:1467] Processing CreateTablet for tablet cb4a2061b8794425bca7fde1072f685a (DEFAULT_TABLE table=test-table [id=fe8cf319116d48e782a042d7db7ad2bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:12.248414 1648 tablet_service.cc:1467] Processing CreateTablet for tablet cb4a2061b8794425bca7fde1072f685a (DEFAULT_TABLE table=test-table [id=fe8cf319116d48e782a042d7db7ad2bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:12.248787 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb4a2061b8794425bca7fde1072f685a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:12.249044 1928 tablet_service.cc:1467] Processing CreateTablet for tablet cb4a2061b8794425bca7fde1072f685a (DEFAULT_TABLE table=test-table [id=fe8cf319116d48e782a042d7db7ad2bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:12.249269 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb4a2061b8794425bca7fde1072f685a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:12.249778 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb4a2061b8794425bca7fde1072f685a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:12.251363 2017 tablet_bootstrap.cc:492] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe: Bootstrap starting.
I20260929 01:02:12.251408 2018 tablet_bootstrap.cc:492] T cb4a2061b8794425bca7fde1072f685a P 8bfba868ebda4a58adbf67edb6b83dc5: Bootstrap starting.
I20260929 01:02:12.251847 2019 tablet_bootstrap.cc:492] T cb4a2061b8794425bca7fde1072f685a P abd6cb9a241b4f44942ee39b5108cd3f: Bootstrap starting.
I20260929 01:02:12.252336 2018 tablet_bootstrap.cc:654] T cb4a2061b8794425bca7fde1072f685a P 8bfba868ebda4a58adbf67edb6b83dc5: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:12.252336 2017 tablet_bootstrap.cc:654] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:12.252715 2019 tablet_bootstrap.cc:654] T cb4a2061b8794425bca7fde1072f685a P abd6cb9a241b4f44942ee39b5108cd3f: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:12.252934 2042 ts_tablet_manager.cc:1918] T 2a19bcf5701d4671be641aed6f26f835 P abd6cb9a241b4f44942ee39b5108cd3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:02:12.253057 2042 log.cc:1196] T 2a19bcf5701d4671be641aed6f26f835 P abd6cb9a241b4f44942ee39b5108cd3f: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-1/wal/wals/2a19bcf5701d4671be641aed6f26f835
I20260929 01:02:12.253372 2042 ts_tablet_manager.cc:1939] T 2a19bcf5701d4671be641aed6f26f835 P abd6cb9a241b4f44942ee39b5108cd3f: Deleting consensus metadata
I20260929 01:02:12.254117 2018 tablet_bootstrap.cc:492] T cb4a2061b8794425bca7fde1072f685a P 8bfba868ebda4a58adbf67edb6b83dc5: No bootstrap required, opened a new log
I20260929 01:02:12.254221 2018 ts_tablet_manager.cc:1397] T cb4a2061b8794425bca7fde1072f685a P 8bfba868ebda4a58adbf67edb6b83dc5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 01:02:12.254457 1495 catalog_manager.cc:4841] TS abd6cb9a241b4f44942ee39b5108cd3f (127.0.98.2:40427): tablet 2a19bcf5701d4671be641aed6f26f835 (table test-table [id=e9c9b13eb36e491a88456ea869dcb3a7]) successfully deleted
I20260929 01:02:12.254693 2018 raft_consensus.cc:348] T cb4a2061b8794425bca7fde1072f685a P 8bfba868ebda4a58adbf67edb6b83dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:12.254850 2018 raft_consensus.cc:374] T cb4a2061b8794425bca7fde1072f685a P 8bfba868ebda4a58adbf67edb6b83dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:12.254890 2018 raft_consensus.cc:729] T cb4a2061b8794425bca7fde1072f685a P 8bfba868ebda4a58adbf67edb6b83dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bfba868ebda4a58adbf67edb6b83dc5, State: Initialized, Role: FOLLOWER
I20260929 01:02:12.255026 2018 consensus_queue.cc:260] T cb4a2061b8794425bca7fde1072f685a P 8bfba868ebda4a58adbf67edb6b83dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:12.255294 2018 ts_tablet_manager.cc:1428] T cb4a2061b8794425bca7fde1072f685a P 8bfba868ebda4a58adbf67edb6b83dc5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 01:02:12.255404 2040 ts_tablet_manager.cc:1905] T 2a19bcf5701d4671be641aed6f26f835 P 8bfba868ebda4a58adbf67edb6b83dc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:12.256152 2017 tablet_bootstrap.cc:492] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe: No bootstrap required, opened a new log
I20260929 01:02:12.256232 2017 ts_tablet_manager.cc:1397] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 01:02:12.256685 2017 raft_consensus.cc:348] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:12.256803 2017 raft_consensus.cc:374] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:12.256839 2017 raft_consensus.cc:729] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 411cd7da2c6248b3991af98fa63c46fe, State: Initialized, Role: FOLLOWER
I20260929 01:02:12.256954 2017 consensus_queue.cc:260] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:12.257234 2017 ts_tablet_manager.cc:1428] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:12.258545 2040 ts_tablet_manager.cc:1918] T 2a19bcf5701d4671be641aed6f26f835 P 8bfba868ebda4a58adbf67edb6b83dc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:02:12.258630 2040 log.cc:1196] T 2a19bcf5701d4671be641aed6f26f835 P 8bfba868ebda4a58adbf67edb6b83dc5: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-0/wal/wals/2a19bcf5701d4671be641aed6f26f835
I20260929 01:02:12.258656 2019 tablet_bootstrap.cc:492] T cb4a2061b8794425bca7fde1072f685a P abd6cb9a241b4f44942ee39b5108cd3f: No bootstrap required, opened a new log
I20260929 01:02:12.258737 2019 ts_tablet_manager.cc:1397] T cb4a2061b8794425bca7fde1072f685a P abd6cb9a241b4f44942ee39b5108cd3f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260929 01:02:12.258941 2040 ts_tablet_manager.cc:1939] T 2a19bcf5701d4671be641aed6f26f835 P 8bfba868ebda4a58adbf67edb6b83dc5: Deleting consensus metadata
I20260929 01:02:12.259250 2019 raft_consensus.cc:348] T cb4a2061b8794425bca7fde1072f685a P abd6cb9a241b4f44942ee39b5108cd3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:12.259364 2019 raft_consensus.cc:374] T cb4a2061b8794425bca7fde1072f685a P abd6cb9a241b4f44942ee39b5108cd3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:12.259397 2019 raft_consensus.cc:729] T cb4a2061b8794425bca7fde1072f685a P abd6cb9a241b4f44942ee39b5108cd3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abd6cb9a241b4f44942ee39b5108cd3f, State: Initialized, Role: FOLLOWER
I20260929 01:02:12.259507 2019 consensus_queue.cc:260] T cb4a2061b8794425bca7fde1072f685a P abd6cb9a241b4f44942ee39b5108cd3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:12.259761 2019 ts_tablet_manager.cc:1428] T cb4a2061b8794425bca7fde1072f685a P abd6cb9a241b4f44942ee39b5108cd3f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:12.260048 1495 catalog_manager.cc:4841] TS 8bfba868ebda4a58adbf67edb6b83dc5 (127.0.98.1:35865): tablet 2a19bcf5701d4671be641aed6f26f835 (table test-table [id=e9c9b13eb36e491a88456ea869dcb3a7]) successfully deleted
I20260929 01:02:12.603636 2035 raft_consensus.cc:482] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:02:12.603780 2035 raft_consensus.cc:504] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:12.604209 2035 leader_election.cc:288] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bfba868ebda4a58adbf67edb6b83dc5 (127.0.98.1:35865), abd6cb9a241b4f44942ee39b5108cd3f (127.0.98.2:40427)
I20260929 01:02:12.604691 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4a2061b8794425bca7fde1072f685a" candidate_uuid: "411cd7da2c6248b3991af98fa63c46fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" is_pre_election: true
I20260929 01:02:12.604770 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4a2061b8794425bca7fde1072f685a" candidate_uuid: "411cd7da2c6248b3991af98fa63c46fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" is_pre_election: true
I20260929 01:02:12.604858 1668 raft_consensus.cc:2454] T cb4a2061b8794425bca7fde1072f685a P 8bfba868ebda4a58adbf67edb6b83dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 411cd7da2c6248b3991af98fa63c46fe in term 0.
I20260929 01:02:12.604950 1808 raft_consensus.cc:2454] T cb4a2061b8794425bca7fde1072f685a P abd6cb9a241b4f44942ee39b5108cd3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 411cd7da2c6248b3991af98fa63c46fe in term 0.
I20260929 01:02:12.605218 1879 leader_election.cc:302] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 411cd7da2c6248b3991af98fa63c46fe, 8bfba868ebda4a58adbf67edb6b83dc5; no voters:
I20260929 01:02:12.605391 2035 raft_consensus.cc:2781] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:02:12.605444 2035 raft_consensus.cc:482] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:02:12.605484 2035 raft_consensus.cc:3037] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:12.606487 2035 raft_consensus.cc:504] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:12.606832 2035 leader_election.cc:288] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe [CANDIDATE]: Term 1 election: Requested vote from peers 8bfba868ebda4a58adbf67edb6b83dc5 (127.0.98.1:35865), abd6cb9a241b4f44942ee39b5108cd3f (127.0.98.2:40427)
I20260929 01:02:12.607165 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4a2061b8794425bca7fde1072f685a" candidate_uuid: "411cd7da2c6248b3991af98fa63c46fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bfba868ebda4a58adbf67edb6b83dc5"
I20260929 01:02:12.607244 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb4a2061b8794425bca7fde1072f685a" candidate_uuid: "411cd7da2c6248b3991af98fa63c46fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abd6cb9a241b4f44942ee39b5108cd3f"
I20260929 01:02:12.607270 1668 raft_consensus.cc:3037] T cb4a2061b8794425bca7fde1072f685a P 8bfba868ebda4a58adbf67edb6b83dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:12.607358 1808 raft_consensus.cc:3037] T cb4a2061b8794425bca7fde1072f685a P abd6cb9a241b4f44942ee39b5108cd3f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:12.608148 1668 raft_consensus.cc:2454] T cb4a2061b8794425bca7fde1072f685a P 8bfba868ebda4a58adbf67edb6b83dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 411cd7da2c6248b3991af98fa63c46fe in term 1.
I20260929 01:02:12.608310 1808 raft_consensus.cc:2454] T cb4a2061b8794425bca7fde1072f685a P abd6cb9a241b4f44942ee39b5108cd3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 411cd7da2c6248b3991af98fa63c46fe in term 1.
I20260929 01:02:12.608431 1879 leader_election.cc:302] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 411cd7da2c6248b3991af98fa63c46fe, 8bfba868ebda4a58adbf67edb6b83dc5; no voters:
I20260929 01:02:12.608640 2035 raft_consensus.cc:2781] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:12.608734 2035 raft_consensus.cc:686] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe [term 1 LEADER]: Becoming Leader. State: Replica: 411cd7da2c6248b3991af98fa63c46fe, State: Running, Role: LEADER
I20260929 01:02:12.608871 2035 consensus_queue.cc:237] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:12.610672 1509 catalog_manager.cc:5495] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 411cd7da2c6248b3991af98fa63c46fe (127.0.98.3). New cstate: current_term: 1 leader_uuid: "411cd7da2c6248b3991af98fa63c46fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } health_report { overall_health: HEALTHY } } }
Sep 29 01:02:12 dist-test-slave-7zdl krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643731, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:12.652949 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:12.644617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43845 (local address 127.0.0.1:34968)
0929 01:02:12.644926 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:12.644941 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:12.645069 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:12.645336 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:12.645338 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:12.645740 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:12.645834 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:12.645839 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.646635 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.646638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:12.647209 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:12.647216 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.647613 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.647615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:12.647673 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:12.648436 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:12.648448 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:12.649880 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:12.651589 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:12.651597 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:12.651602 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:12.651819 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:12.652055 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:12.652057 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:12.652058 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:12.652130 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:12.652440 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:12.652446 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:12.652548 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 01:02:12.652637 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 01:02:12.652772 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":102,"threads_started":1}
I20260929 01:02:12.652956 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:12.644689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34968 (local address 127.0.98.62:43845)
0929 01:02:12.644953 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:12.644960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:12.645002 (+ 42us) server_negotiation.cc:407] Connection header received
0929 01:02:12.645169 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:12.645171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:12.645212 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:12.645315 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:12.645943 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.646536 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.647308 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.647517 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.647593 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:12.649996 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:12.649999 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:12.650001 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:12.650021 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:12.651498 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:12.651903 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:12.651906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:12.651907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:12.651979 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:12.652203 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:12.652205 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:12.652207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:12.652357 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:12.652428 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:12.652692 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 01:02:12.652828 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":190,"thread_start_us":107,"threads_started":1}
I20260929 01:02:12.656540 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34968:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 01:02:12.657270 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:02:12.661768 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 0e1bcf09176749f4a7a496bdf0d6911c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fcbbf9d9ee65458bb2249aa9c9d0f78c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:02:12.661890 1788 tablet_service.cc:1467] Processing CreateTablet for tablet 0e1bcf09176749f4a7a496bdf0d6911c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fcbbf9d9ee65458bb2249aa9c9d0f78c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:02:12.661963 1928 tablet_service.cc:1467] Processing CreateTablet for tablet 0e1bcf09176749f4a7a496bdf0d6911c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fcbbf9d9ee65458bb2249aa9c9d0f78c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:02:12.662176 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e1bcf09176749f4a7a496bdf0d6911c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:12.662274 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e1bcf09176749f4a7a496bdf0d6911c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:12.662317 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e1bcf09176749f4a7a496bdf0d6911c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:12.664867 2018 tablet_bootstrap.cc:492] T 0e1bcf09176749f4a7a496bdf0d6911c P 8bfba868ebda4a58adbf67edb6b83dc5: Bootstrap starting.
I20260929 01:02:12.664942 2017 tablet_bootstrap.cc:492] T 0e1bcf09176749f4a7a496bdf0d6911c P 411cd7da2c6248b3991af98fa63c46fe: Bootstrap starting.
I20260929 01:02:12.665130 2019 tablet_bootstrap.cc:492] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f: Bootstrap starting.
I20260929 01:02:12.665810 2017 tablet_bootstrap.cc:654] T 0e1bcf09176749f4a7a496bdf0d6911c P 411cd7da2c6248b3991af98fa63c46fe: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:12.665812 2018 tablet_bootstrap.cc:654] T 0e1bcf09176749f4a7a496bdf0d6911c P 8bfba868ebda4a58adbf67edb6b83dc5: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:12.666019 2019 tablet_bootstrap.cc:654] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:12.667104 2017 tablet_bootstrap.cc:492] T 0e1bcf09176749f4a7a496bdf0d6911c P 411cd7da2c6248b3991af98fa63c46fe: No bootstrap required, opened a new log
I20260929 01:02:12.667121 2018 tablet_bootstrap.cc:492] T 0e1bcf09176749f4a7a496bdf0d6911c P 8bfba868ebda4a58adbf67edb6b83dc5: No bootstrap required, opened a new log
I20260929 01:02:12.667186 2018 ts_tablet_manager.cc:1397] T 0e1bcf09176749f4a7a496bdf0d6911c P 8bfba868ebda4a58adbf67edb6b83dc5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 01:02:12.667186 2017 ts_tablet_manager.cc:1397] T 0e1bcf09176749f4a7a496bdf0d6911c P 411cd7da2c6248b3991af98fa63c46fe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 01:02:12.667219 2019 tablet_bootstrap.cc:492] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f: No bootstrap required, opened a new log
I20260929 01:02:12.667294 2019 ts_tablet_manager.cc:1397] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:12.667572 2017 raft_consensus.cc:348] T 0e1bcf09176749f4a7a496bdf0d6911c P 411cd7da2c6248b3991af98fa63c46fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:12.667572 2018 raft_consensus.cc:348] T 0e1bcf09176749f4a7a496bdf0d6911c P 8bfba868ebda4a58adbf67edb6b83dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:12.667675 2017 raft_consensus.cc:374] T 0e1bcf09176749f4a7a496bdf0d6911c P 411cd7da2c6248b3991af98fa63c46fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:12.667675 2018 raft_consensus.cc:374] T 0e1bcf09176749f4a7a496bdf0d6911c P 8bfba868ebda4a58adbf67edb6b83dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:12.667711 2017 raft_consensus.cc:729] T 0e1bcf09176749f4a7a496bdf0d6911c P 411cd7da2c6248b3991af98fa63c46fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 411cd7da2c6248b3991af98fa63c46fe, State: Initialized, Role: FOLLOWER
I20260929 01:02:12.667711 2018 raft_consensus.cc:729] T 0e1bcf09176749f4a7a496bdf0d6911c P 8bfba868ebda4a58adbf67edb6b83dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bfba868ebda4a58adbf67edb6b83dc5, State: Initialized, Role: FOLLOWER
I20260929 01:02:12.667716 2019 raft_consensus.cc:348] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:12.667841 2019 raft_consensus.cc:374] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:12.667835 2017 consensus_queue.cc:260] T 0e1bcf09176749f4a7a496bdf0d6911c P 411cd7da2c6248b3991af98fa63c46fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:12.667835 2018 consensus_queue.cc:260] T 0e1bcf09176749f4a7a496bdf0d6911c P 8bfba868ebda4a58adbf67edb6b83dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:12.667879 2019 raft_consensus.cc:729] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abd6cb9a241b4f44942ee39b5108cd3f, State: Initialized, Role: FOLLOWER
I20260929 01:02:12.667999 2019 consensus_queue.cc:260] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:12.668094 2017 ts_tablet_manager.cc:1428] T 0e1bcf09176749f4a7a496bdf0d6911c P 411cd7da2c6248b3991af98fa63c46fe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 01:02:12.668094 2018 ts_tablet_manager.cc:1428] T 0e1bcf09176749f4a7a496bdf0d6911c P 8bfba868ebda4a58adbf67edb6b83dc5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 01:02:12.668054 2024 tablet_replica.cc:439] T 0e1bcf09176749f4a7a496bdf0d6911c P 8bfba868ebda4a58adbf67edb6b83dc5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } } }
I20260929 01:02:12.668185 2024 tablet_replica.cc:442] T 0e1bcf09176749f4a7a496bdf0d6911c P 8bfba868ebda4a58adbf67edb6b83dc5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:02:12.668267 2019 ts_tablet_manager.cc:1428] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:12.668363 2035 tablet_replica.cc:439] T 0e1bcf09176749f4a7a496bdf0d6911c P 411cd7da2c6248b3991af98fa63c46fe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } } }
I20260929 01:02:12.668476 2035 tablet_replica.cc:442] T 0e1bcf09176749f4a7a496bdf0d6911c P 411cd7da2c6248b3991af98fa63c46fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:02:12.668464 2025 tablet_replica.cc:439] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } } }
I20260929 01:02:12.668586 2025 tablet_replica.cc:442] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:02:12.671732 2025 raft_consensus.cc:482] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:02:12.671825 2025 raft_consensus.cc:504] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:12.672827 2025 leader_election.cc:288] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bfba868ebda4a58adbf67edb6b83dc5 (127.0.98.1:35865), 411cd7da2c6248b3991af98fa63c46fe (127.0.98.3:45107)
I20260929 01:02:12.676662 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:12.673041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43603 (local address 127.0.98.1:35865)
0929 01:02:12.673264 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:12.673269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:12.673286 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:02:12.673371 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:12.673374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:12.673423 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:12.673504 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:12.673939 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.674489 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.675662 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.676088 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.676213 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:12.676273 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:12.676476 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 01:02:12.676545 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":50,"threads_started":1}
I20260929 01:02:12.676673 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:12.672960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35865 (local address 127.0.98.2:43603)
0929 01:02:12.673163 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:12.673180 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:12.673277 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:12.673647 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:12.673650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:12.673663 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:12.673780 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:12.673786 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.674598 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.674603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:12.675527 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:12.675536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.676226 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.676230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:12.676315 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:12.676362 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 01:02:12.676444 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 01:02:12.676515 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":2,"thread_start_us":60,"threads_started":1}
I20260929 01:02:12.676700 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:12.673392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48193 (local address 127.0.98.3:45107)
0929 01:02:12.673486 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:12.673491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:12.673593 (+ 102us) server_negotiation.cc:407] Connection header received
0929 01:02:12.673774 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:12.673777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:12.673826 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:12.673898 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:12.674163 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.674652 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.675695 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.676081 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.676201 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:12.676278 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:12.676528 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 01:02:12.676597 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 01:02:12.676719 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:12.673051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45107 (local address 127.0.98.2:48193)
0929 01:02:12.673514 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:12.673528 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:12.673651 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:12.673920 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:12.673923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:12.673939 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:12.674059 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:12.674064 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.674763 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.674769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:12.675567 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:12.675575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.676250 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.676254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:12.676389 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:12.676422 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 01:02:12.676512 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 01:02:12.676582 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":30,"thread_start_us":110,"threads_started":1}
I20260929 01:02:12.677425 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e1bcf09176749f4a7a496bdf0d6911c" candidate_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" is_pre_election: true
I20260929 01:02:12.677574 1668 raft_consensus.cc:2454] T 0e1bcf09176749f4a7a496bdf0d6911c P 8bfba868ebda4a58adbf67edb6b83dc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abd6cb9a241b4f44942ee39b5108cd3f in term 0.
I20260929 01:02:12.677791 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e1bcf09176749f4a7a496bdf0d6911c" candidate_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "411cd7da2c6248b3991af98fa63c46fe" is_pre_election: true
I20260929 01:02:12.677917 1739 leader_election.cc:302] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bfba868ebda4a58adbf67edb6b83dc5, abd6cb9a241b4f44942ee39b5108cd3f; no voters:
I20260929 01:02:12.677968 1948 raft_consensus.cc:2454] T 0e1bcf09176749f4a7a496bdf0d6911c P 411cd7da2c6248b3991af98fa63c46fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abd6cb9a241b4f44942ee39b5108cd3f in term 0.
I20260929 01:02:12.678141 2025 raft_consensus.cc:2781] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:02:12.678205 2025 raft_consensus.cc:482] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:02:12.678252 2025 raft_consensus.cc:3037] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:12.679224 2025 raft_consensus.cc:504] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:12.679543 2025 leader_election.cc:288] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f [CANDIDATE]: Term 1 election: Requested vote from peers 8bfba868ebda4a58adbf67edb6b83dc5 (127.0.98.1:35865), 411cd7da2c6248b3991af98fa63c46fe (127.0.98.3:45107)
I20260929 01:02:12.679857 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e1bcf09176749f4a7a496bdf0d6911c" candidate_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "411cd7da2c6248b3991af98fa63c46fe"
I20260929 01:02:12.679873 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e1bcf09176749f4a7a496bdf0d6911c" candidate_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bfba868ebda4a58adbf67edb6b83dc5"
I20260929 01:02:12.679983 1668 raft_consensus.cc:3037] T 0e1bcf09176749f4a7a496bdf0d6911c P 8bfba868ebda4a58adbf67edb6b83dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:12.679983 1948 raft_consensus.cc:3037] T 0e1bcf09176749f4a7a496bdf0d6911c P 411cd7da2c6248b3991af98fa63c46fe [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:12.681088 1668 raft_consensus.cc:2454] T 0e1bcf09176749f4a7a496bdf0d6911c P 8bfba868ebda4a58adbf67edb6b83dc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abd6cb9a241b4f44942ee39b5108cd3f in term 1.
I20260929 01:02:12.681088 1948 raft_consensus.cc:2454] T 0e1bcf09176749f4a7a496bdf0d6911c P 411cd7da2c6248b3991af98fa63c46fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abd6cb9a241b4f44942ee39b5108cd3f in term 1.
I20260929 01:02:12.681427 1741 leader_election.cc:302] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 411cd7da2c6248b3991af98fa63c46fe, abd6cb9a241b4f44942ee39b5108cd3f; no voters:
I20260929 01:02:12.681587 2025 raft_consensus.cc:2781] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:12.681803 2025 raft_consensus.cc:686] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f [term 1 LEADER]: Becoming Leader. State: Replica: abd6cb9a241b4f44942ee39b5108cd3f, State: Running, Role: LEADER
I20260929 01:02:12.682049 2025 consensus_queue.cc:237] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } }
I20260929 01:02:12.683202 2060 tablet_replica.cc:439] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f: TxnStatusTablet state changed. Reason: New leader abd6cb9a241b4f44942ee39b5108cd3f. Latest consensus state: current_term: 1 leader_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } } }
I20260929 01:02:12.683300 2060 tablet_replica.cc:442] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:02:12.683564 2062 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 01:02:12.683707 2062 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 01:02:12.684423 1511 catalog_manager.cc:5495] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f reported cstate change: term changed from 0 to 1, leader changed from <none> to abd6cb9a241b4f44942ee39b5108cd3f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } health_report { overall_health: UNKNOWN } } }
I20260929 01:02:12.697299 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:12.693341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37412 (local address 127.0.98.2:40427)
0929 01:02:12.693596 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:12.693602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:12.693613 (+ 11us) server_negotiation.cc:407] Connection header received
0929 01:02:12.693689 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:12.693692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:12.693736 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:12.693826 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:02:12.694129 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.694775 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.695973 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.696181 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.696253 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:12.696586 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:02:12.696971 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:02:12.697045 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:12.697152 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 01:02:12.697193 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":95,"threads_started":1}
I20260929 01:02:12.697333 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:12.693206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40427 (local address 127.0.0.1:37412)
0929 01:02:12.693419 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:12.693435 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:12.693539 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:12.693883 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:12.693901 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:12.693912 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:02:12.694024 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:12.694031 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.694908 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.694911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:12.695866 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:12.695876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.696280 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.696284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:12.696370 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:12.696418 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:02:12.697054 (+ 636us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:02:12.697067 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 01:02:12.697136 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 01:02:12.697213 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260929 01:02:12.700045 1668 raft_consensus.cc:1261] T 0e1bcf09176749f4a7a496bdf0d6911c P 8bfba868ebda4a58adbf67edb6b83dc5 [term 1 FOLLOWER]: Refusing update from remote peer abd6cb9a241b4f44942ee39b5108cd3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:12.700541 2025 consensus_queue.cc:1035] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:12.700618 1948 raft_consensus.cc:1261] T 0e1bcf09176749f4a7a496bdf0d6911c P 411cd7da2c6248b3991af98fa63c46fe [term 1 FOLLOWER]: Refusing update from remote peer abd6cb9a241b4f44942ee39b5108cd3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:12.701090 2025 consensus_queue.cc:1035] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:12.702312 2024 tablet_replica.cc:439] T 0e1bcf09176749f4a7a496bdf0d6911c P 8bfba868ebda4a58adbf67edb6b83dc5: TxnStatusTablet state changed. Reason: New leader abd6cb9a241b4f44942ee39b5108cd3f. Latest consensus state: current_term: 1 leader_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } } }
I20260929 01:02:12.702422 2024 tablet_replica.cc:442] T 0e1bcf09176749f4a7a496bdf0d6911c P 8bfba868ebda4a58adbf67edb6b83dc5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:02:12.703539 2035 tablet_replica.cc:439] T 0e1bcf09176749f4a7a496bdf0d6911c P 411cd7da2c6248b3991af98fa63c46fe: TxnStatusTablet state changed. Reason: New leader abd6cb9a241b4f44942ee39b5108cd3f. Latest consensus state: current_term: 1 leader_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } } }
I20260929 01:02:12.703637 2035 tablet_replica.cc:442] T 0e1bcf09176749f4a7a496bdf0d6911c P 411cd7da2c6248b3991af98fa63c46fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:02:12.704854 2060 tablet_replica.cc:439] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } } }
I20260929 01:02:12.704955 2060 tablet_replica.cc:442] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:02:12.705026 2068 tablet_replica.cc:439] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } } }
I20260929 01:02:12.705128 2068 tablet_replica.cc:442] T 0e1bcf09176749f4a7a496bdf0d6911c P abd6cb9a241b4f44942ee39b5108cd3f: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:02:12.705399 2035 tablet_replica.cc:439] T 0e1bcf09176749f4a7a496bdf0d6911c P 411cd7da2c6248b3991af98fa63c46fe: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } } }
I20260929 01:02:12.705506 2035 tablet_replica.cc:442] T 0e1bcf09176749f4a7a496bdf0d6911c P 411cd7da2c6248b3991af98fa63c46fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:02:12.711687 2032 mvcc.cc:205] Tried to move back new op lower bound from 7334476729134772224 to 7334476729068085248. Current Snapshot: MvccSnapshot[applied={T|T < 7334476729134772224}]
I20260929 01:02:12.711871 2024 tablet_replica.cc:439] T 0e1bcf09176749f4a7a496bdf0d6911c P 8bfba868ebda4a58adbf67edb6b83dc5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 } } peers { permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 } } peers { permanent_uuid: "411cd7da2c6248b3991af98fa63c46fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45107 } } }
I20260929 01:02:12.711994 2024 tablet_replica.cc:442] T 0e1bcf09176749f4a7a496bdf0d6911c P 8bfba868ebda4a58adbf67edb6b83dc5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:02:12.725458 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:12.721636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54229 (local address 127.0.98.2:40427)
0929 01:02:12.721738 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:12.721742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:12.721756 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:02:12.721807 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:12.721812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:12.721861 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:12.721951 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:02:12.722178 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.723029 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.724283 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.724632 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.724734 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:12.725026 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:02:12.725104 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:02:12.725175 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:12.725285 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 01:02:12.725337 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 01:02:12.725478 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:12.721312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40427 (local address 127.0.98.3:54229)
0929 01:02:12.721616 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:12.721632 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:12.721730 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:12.721961 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:12.721964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:12.721973 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:02:12.722072 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:12.722078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.723144 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.723148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:12.724168 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:12.724177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.724747 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.724751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:12.724835 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:12.724881 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:02:12.725191 (+ 310us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:02:12.725197 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 01:02:12.725258 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 01:02:12.725336 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":99,"threads_started":1}
I20260929 01:02:12.732312 1668 raft_consensus.cc:1261] T cb4a2061b8794425bca7fde1072f685a P 8bfba868ebda4a58adbf67edb6b83dc5 [term 1 FOLLOWER]: Refusing update from remote peer 411cd7da2c6248b3991af98fa63c46fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:12.732331 1808 raft_consensus.cc:1261] T cb4a2061b8794425bca7fde1072f685a P abd6cb9a241b4f44942ee39b5108cd3f [term 1 FOLLOWER]: Refusing update from remote peer 411cd7da2c6248b3991af98fa63c46fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:12.732710 2023 consensus_queue.cc:1035] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bfba868ebda4a58adbf67edb6b83dc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:12.733003 2023 consensus_queue.cc:1035] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "abd6cb9a241b4f44942ee39b5108cd3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:12.735525 2030 mvcc.cc:205] Tried to move back new op lower bound from 7334476729267486720 to 7334476728767586304. Current Snapshot: MvccSnapshot[applied={T|T < 7334476729267486720}]
I20260929 01:02:12.755956 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:12.751860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46141 (local address 127.0.98.3:45107)
0929 01:02:12.751993 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:12.751999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:12.752466 (+ 467us) server_negotiation.cc:407] Connection header received
0929 01:02:12.752632 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:12.752635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:12.752687 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:12.752768 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:02:12.753018 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.753772 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.754717 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.755026 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.755133 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:12.755480 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:02:12.755560 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:02:12.755630 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:12.755766 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 01:02:12.755817 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 01:02:12.755978 2075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:12.751795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45107 (local address 127.0.98.2:46141)
0929 01:02:12.752368 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:12.752380 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:12.752543 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:12.752775 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:12.752778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:12.752785 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:02:12.752926 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:12.752937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.753874 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.753879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:12.754593 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:12.754603 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.755131 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.755135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:12.755236 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:12.755322 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:02:12.755648 (+ 326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:02:12.755656 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 01:02:12.755744 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 01:02:12.755818 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":104,"threads_started":1}
I20260929 01:02:12.777323 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:12.777532 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:12.779397 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47695d2b50a2404ebc1671f8e6652f56: Sending DeleteTablet for 3 replicas of tablet cb4a2061b8794425bca7fde1072f685a
I20260929 01:02:12.779912 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet cb4a2061b8794425bca7fde1072f685a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36160
I20260929 01:02:12.779999 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet cb4a2061b8794425bca7fde1072f685a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37400
I20260929 01:02:12.780164 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet cb4a2061b8794425bca7fde1072f685a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47048
I20260929 01:02:12.780295 2078 tablet_replica.cc:330] T cb4a2061b8794425bca7fde1072f685a P abd6cb9a241b4f44942ee39b5108cd3f: stopping tablet replica
I20260929 01:02:12.780409 2079 tablet_replica.cc:330] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe: stopping tablet replica
I20260929 01:02:12.780445 2078 raft_consensus.cc:2229] T cb4a2061b8794425bca7fde1072f685a P abd6cb9a241b4f44942ee39b5108cd3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:12.780521 2079 raft_consensus.cc:2229] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:02:12.780557 2078 raft_consensus.cc:2258] T cb4a2061b8794425bca7fde1072f685a P abd6cb9a241b4f44942ee39b5108cd3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:12.780690 2079 raft_consensus.cc:2258] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:12.781136 2077 tablet_replica.cc:330] T cb4a2061b8794425bca7fde1072f685a P 8bfba868ebda4a58adbf67edb6b83dc5: stopping tablet replica
I20260929 01:02:12.781296 2077 raft_consensus.cc:2229] T cb4a2061b8794425bca7fde1072f685a P 8bfba868ebda4a58adbf67edb6b83dc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:12.781311 2079 ts_tablet_manager.cc:1905] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:12.781486 2077 raft_consensus.cc:2258] T cb4a2061b8794425bca7fde1072f685a P 8bfba868ebda4a58adbf67edb6b83dc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:12.782063 2078 ts_tablet_manager.cc:1905] T cb4a2061b8794425bca7fde1072f685a P abd6cb9a241b4f44942ee39b5108cd3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:12.782184 2077 ts_tablet_manager.cc:1905] T cb4a2061b8794425bca7fde1072f685a P 8bfba868ebda4a58adbf67edb6b83dc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:12.784161 2079 ts_tablet_manager.cc:1918] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:02:12.784240 2079 log.cc:1196] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/cb4a2061b8794425bca7fde1072f685a
I20260929 01:02:12.784544 2079 ts_tablet_manager.cc:1939] T cb4a2061b8794425bca7fde1072f685a P 411cd7da2c6248b3991af98fa63c46fe: Deleting consensus metadata
I20260929 01:02:12.784873 2078 ts_tablet_manager.cc:1918] T cb4a2061b8794425bca7fde1072f685a P abd6cb9a241b4f44942ee39b5108cd3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:02:12.784962 2078 log.cc:1196] T cb4a2061b8794425bca7fde1072f685a P abd6cb9a241b4f44942ee39b5108cd3f: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-1/wal/wals/cb4a2061b8794425bca7fde1072f685a
I20260929 01:02:12.785166 1497 catalog_manager.cc:4841] TS 411cd7da2c6248b3991af98fa63c46fe (127.0.98.3:45107): tablet cb4a2061b8794425bca7fde1072f685a (table test-table [id=fe8cf319116d48e782a042d7db7ad2bc]) successfully deleted
I20260929 01:02:12.785209 2078 ts_tablet_manager.cc:1939] T cb4a2061b8794425bca7fde1072f685a P abd6cb9a241b4f44942ee39b5108cd3f: Deleting consensus metadata
I20260929 01:02:12.785189 2077 ts_tablet_manager.cc:1918] T cb4a2061b8794425bca7fde1072f685a P 8bfba868ebda4a58adbf67edb6b83dc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:02:12.785359 2077 log.cc:1196] T cb4a2061b8794425bca7fde1072f685a P 8bfba868ebda4a58adbf67edb6b83dc5: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790643720886468-392-0/minicluster-data/ts-0/wal/wals/cb4a2061b8794425bca7fde1072f685a
I20260929 01:02:12.785642 2077 ts_tablet_manager.cc:1939] T cb4a2061b8794425bca7fde1072f685a P 8bfba868ebda4a58adbf67edb6b83dc5: Deleting consensus metadata
I20260929 01:02:12.785858 1495 catalog_manager.cc:4841] TS abd6cb9a241b4f44942ee39b5108cd3f (127.0.98.2:40427): tablet cb4a2061b8794425bca7fde1072f685a (table test-table [id=fe8cf319116d48e782a042d7db7ad2bc]) successfully deleted
I20260929 01:02:12.786283 1495 catalog_manager.cc:4841] TS 8bfba868ebda4a58adbf67edb6b83dc5 (127.0.98.1:35865): tablet cb4a2061b8794425bca7fde1072f685a (table test-table [id=fe8cf319116d48e782a042d7db7ad2bc]) successfully deleted
Sep 29 01:02:12 dist-test-slave-7zdl krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643731, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 01:02:12.795740 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:12.786184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36170 (local address 127.0.98.1:35865)
0929 01:02:12.786286 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:12.786291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:12.786521 (+ 230us) server_negotiation.cc:407] Connection header received
0929 01:02:12.786633 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:12.786636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:12.786682 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:12.786754 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:12.787442 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.788225 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.788988 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.789308 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.789405 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:12.791973 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:12.791983 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:12.792000 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:12.792037 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:12.793970 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:12.794384 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:12.794398 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:12.794400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:12.794485 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:12.794769 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:12.794772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:12.794774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:12.795088 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:12.795301 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:12.795457 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 01:02:12.795575 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260929 01:02:12.796200 1708 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36170
I20260929 01:02:12.805344 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:12.797842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34970 (local address 127.0.98.62:43845)
0929 01:02:12.797960 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:12.797966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:12.798057 (+ 91us) server_negotiation.cc:407] Connection header received
0929 01:02:12.798182 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:12.798184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:12.798225 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:12.798288 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:12.799032 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.799560 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.800602 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:12.800813 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:12.800886 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:12.802319 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:12.802322 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:12.802324 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:12.802347 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:12.803758 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:12.804240 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:12.804242 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:12.804244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:12.804309 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:12.804577 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:12.804579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:12.804580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:12.804732 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:12.804829 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:12.805083 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 01:02:12.805178 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 01:02:12.805737 1509 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34970
I20260929 01:02:12.806854 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 1583
I20260929 01:02:12.813586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 1723
I20260929 01:02:12.820341 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 1863
I20260929 01:02:12.826754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 1479
2026-09-29T01:02:12Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4284 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:02:12 dist-test-slave-7zdl krb5kdc[2086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:02:12 dist-test-slave-7zdl krb5kdc[2086](info): set up 2 sockets
Sep 29 01:02:12 dist-test-slave-7zdl krb5kdc[2086](info): commencing operation
krb5kdc: starting...
W20260929 01:02:13.878737 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:02:13 dist-test-slave-7zdl krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:02:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:02:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:02:13.997689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39297
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:14.103894 2099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:14.104203 2099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:14.104285 2099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:14.108249 2099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:02:14.108330 2099 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 01:02:14.108362 2099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:14.108418 2099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:14.108469 2099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:02:14.108508 2099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:02:14.113682 2099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39297
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:14.114710 2099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:14.115728 2099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:14.123940 2104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:14.123945 2107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:14.124075 2105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:14.124503 2099 server_base.cc:1034] running on GCE node
I20260929 01:02:14.125023 2099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:14.126096 2099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:14.127276 2099 hybrid_clock.cc:648] HybridClock initialized: now 1790643734127264 us; error 25 us; skew 500 ppm
Sep 29 01:02:14 dist-test-slave-7zdl krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:14.129849 2099 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:02:14.130721 2099 webserver.cc:460] Webserver started at http://127.0.98.62:44703/ using document root <none> and password file <none>
I20260929 01:02:14.131317 2099 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:14.131404 2099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:14.131636 2099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:14.133591 2099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "f1344acb25074603a231c2ab0d2ed0e3"
format_stamp: "Formatted at 2026-09-29 01:02:14 on dist-test-slave-7zdl"
server_key: "6a044a62a31fbb4f4027ca4578d71cba"
server_key_iv: "cb9572dac449124b45b48a276e7ec0a6"
server_key_version: "encryptionkey@0"
I20260929 01:02:14.134132 2099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "f1344acb25074603a231c2ab0d2ed0e3"
format_stamp: "Formatted at 2026-09-29 01:02:14 on dist-test-slave-7zdl"
server_key: "6a044a62a31fbb4f4027ca4578d71cba"
server_key_iv: "cb9572dac449124b45b48a276e7ec0a6"
server_key_version: "encryptionkey@0"
I20260929 01:02:14.137665 2099 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 01:02:14.139936 2114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:14.141065 2099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:02:14.141201 2099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "f1344acb25074603a231c2ab0d2ed0e3"
format_stamp: "Formatted at 2026-09-29 01:02:14 on dist-test-slave-7zdl"
server_key: "6a044a62a31fbb4f4027ca4578d71cba"
server_key_iv: "cb9572dac449124b45b48a276e7ec0a6"
server_key_version: "encryptionkey@0"
I20260929 01:02:14.141306 2099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:14.153641 2099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:14.156584 2099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:14.156776 2099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:14.164708 2099 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39297
I20260929 01:02:14.164729 2176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39297 every 8 connection(s)
I20260929 01:02:14.165638 2099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:02:14.168306 2177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:14.173375 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 2099
I20260929 01:02:14.173486 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:02:14.173766 392 external_mini_cluster.cc:1428] Setting key 402e6048893591656a0de06f52fd3690
I20260929 01:02:14.173820 2177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3: Bootstrap starting.
I20260929 01:02:14.176295 2177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:14.177021 2177 log.cc:824] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:14.178872 2177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3: No bootstrap required, opened a new log
Sep 29 01:02:14 dist-test-slave-7zdl krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:14.182893 2177 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1344acb25074603a231c2ab0d2ed0e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39297 } }
I20260929 01:02:14.183161 2177 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:14.183279 2177 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1344acb25074603a231c2ab0d2ed0e3, State: Initialized, Role: FOLLOWER
I20260929 01:02:14.183795 2177 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1344acb25074603a231c2ab0d2ed0e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39297 } }
I20260929 01:02:14.183969 2177 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:02:14.184067 2177 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:02:14.184193 2177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:14.185139 2177 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1344acb25074603a231c2ab0d2ed0e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39297 } }
I20260929 01:02:14.185472 2177 leader_election.cc:302] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1344acb25074603a231c2ab0d2ed0e3; no voters:
I20260929 01:02:14.185820 2177 leader_election.cc:288] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:02:14.185916 2182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:14.186156 2182 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3 [term 1 LEADER]: Becoming Leader. State: Replica: f1344acb25074603a231c2ab0d2ed0e3, State: Running, Role: LEADER
I20260929 01:02:14.186440 2182 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1344acb25074603a231c2ab0d2ed0e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39297 } }
I20260929 01:02:14.187206 2177 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:02:14.188305 2184 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1344acb25074603a231c2ab0d2ed0e3. Latest consensus state: current_term: 1 leader_uuid: "f1344acb25074603a231c2ab0d2ed0e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1344acb25074603a231c2ab0d2ed0e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39297 } } }
I20260929 01:02:14.188442 2184 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:14.188585 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:14.175161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47682 (local address 127.0.98.62:39297)
0929 01:02:14.175598 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:14.175613 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:14.175648 (+ 35us) server_negotiation.cc:407] Connection header received
0929 01:02:14.176268 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:14.176284 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:14.176666 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:14.177012 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:14.177768 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.178716 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.179691 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.180161 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.180322 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:14.183379 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:14.183386 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:14.183408 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:14.183433 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:14.185225 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:14.185688 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:14.185695 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:14.185700 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:14.185845 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:14.186194 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:14.186197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:14.186199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:14.186735 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:14.186967 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:14.187227 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 01:02:14.187552 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":136,"threads_started":1}
I20260929 01:02:14.188769 2183 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1344acb25074603a231c2ab0d2ed0e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1344acb25074603a231c2ab0d2ed0e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39297 } } }
I20260929 01:02:14.188843 2183 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:14.189181 2191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:02:14.192153 2191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:02:14.197361 2191 catalog_manager.cc:1347] Generated new cluster ID: d589674a0f2642e2aba4ec799ad456eb
I20260929 01:02:14.197424 2191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:02:14.207602 2191 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:02:14.208494 2191 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:02:14.213439 2191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3: Generated new TSK 0
I20260929 01:02:14.214177 2191 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 01:02:14.250486 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39297
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:14.358739 2203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:14.359102 2203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:14.359205 2203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:14.363212 2203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:14.363302 2203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:14.363387 2203 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 01:02:14.363461 2203 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:02:14.363536 2203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:02:14.368762 2203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:14.369805 2203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:14.370878 2203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:14.380484 2209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:14.380537 2208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:14.380491 2211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:14.380687 2203 server_base.cc:1034] running on GCE node
I20260929 01:02:14.381083 2203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:14.381609 2203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:14.382865 2203 hybrid_clock.cc:648] HybridClock initialized: now 1790643734382833 us; error 41 us; skew 500 ppm
Sep 29 01:02:14 dist-test-slave-7zdl krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:14.385514 2203 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:02:14.386336 2203 webserver.cc:460] Webserver started at http://127.0.98.1:41337/ using document root <none> and password file <none>
I20260929 01:02:14.386992 2203 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:14.387094 2203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:14.387300 2203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:14.389624 2203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "2ac54ced6dad46969c9b046507046048"
format_stamp: "Formatted at 2026-09-29 01:02:14 on dist-test-slave-7zdl"
server_key: "7c9a821e5a7c0c0968980c549395fe85"
server_key_iv: "6ac23191552d282587886446c7ba695d"
server_key_version: "encryptionkey@0"
I20260929 01:02:14.390165 2203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2ac54ced6dad46969c9b046507046048"
format_stamp: "Formatted at 2026-09-29 01:02:14 on dist-test-slave-7zdl"
server_key: "7c9a821e5a7c0c0968980c549395fe85"
server_key_iv: "6ac23191552d282587886446c7ba695d"
server_key_version: "encryptionkey@0"
I20260929 01:02:14.393667 2203 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 01:02:14.396008 2218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:14.397113 2203 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:02:14.397243 2203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "2ac54ced6dad46969c9b046507046048"
format_stamp: "Formatted at 2026-09-29 01:02:14 on dist-test-slave-7zdl"
server_key: "7c9a821e5a7c0c0968980c549395fe85"
server_key_iv: "6ac23191552d282587886446c7ba695d"
server_key_version: "encryptionkey@0"
I20260929 01:02:14.397344 2203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:14.418582 2203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:14.421476 2203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:14.421692 2203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:14.423171 2203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:14.423245 2203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:14.423318 2203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:14.423360 2203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:02:14 dist-test-slave-7zdl krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:14.434355 2203 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45525
I20260929 01:02:14.434474 2333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45525 every 8 connection(s)
I20260929 01:02:14.435297 2203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:02:14.436553 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 2203
I20260929 01:02:14.436638 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:02:14.436924 392 external_mini_cluster.cc:1428] Setting key 56b0a8347056262342b2267eb9bfd4af
I20260929 01:02:14.437223 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:14.423991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49821 (local address 127.0.98.62:39297)
0929 01:02:14.424133 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:14.424137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:14.425149 (+ 1012us) server_negotiation.cc:407] Connection header received
0929 01:02:14.426259 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:14.426265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:14.426330 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:14.426391 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:14.427804 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.428716 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.430014 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.430473 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.430592 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:14.433368 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:14.433372 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:14.433375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:14.433401 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:14.435064 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:14.435632 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:14.435637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:14.435639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:14.435743 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:14.436090 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:14.436095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:14.436097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:14.436272 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:14.436381 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:14.436930 (+ 549us) server_negotiation.cc:299] Negotiation successful
0929 01:02:14.437049 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 01:02:14.438047 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:14.424534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39297 (local address 127.0.98.1:49821)
0929 01:02:14.425005 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:14.425045 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:14.426007 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:14.426863 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:14.426873 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:14.427211 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:14.427653 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:14.427665 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.428854 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.428859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:14.429890 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:14.429899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.430723 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.430727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:14.430936 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:14.431654 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:14.431669 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:14.433227 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:14.435208 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:14.435214 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:14.435228 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:14.435535 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:14.435852 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:14.435855 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:14.435856 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:14.436006 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:14.436396 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:14.436403 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:14.436656 (+ 253us) client_negotiation.cc:769] Sending connection context
0929 01:02:14.436889 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 01:02:14.437178 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":135,"threads_started":1}
I20260929 01:02:14.439318 2334 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39297
I20260929 01:02:14.439465 2334 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:14.440021 2334 heartbeater.cc:511] Master 127.0.98.62:39297 requested a full tablet report, sending...
I20260929 01:02:14.441534 2130 ts_manager.cc:194] Registered new tserver with Master: 2ac54ced6dad46969c9b046507046048 (127.0.98.1:45525)
I20260929 01:02:14.442656 2130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49821
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 01:02:14.451793 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:14.441531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49031 (local address 127.0.98.62:39297)
0929 01:02:14.441650 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:14.441653 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:14.444655 (+ 3002us) server_negotiation.cc:407] Connection header received
0929 01:02:14.444890 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:14.444894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:14.444948 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:14.444998 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:14.445738 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.446260 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.447206 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.447418 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.447517 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:14.448605 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:14.448608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:14.448611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:14.448633 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:14.450100 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:14.450565 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:14.450570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:14.450572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:14.450645 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:14.450995 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:14.451000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:14.451002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:14.451165 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:14.451268 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:14.451530 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 01:02:14.451632 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 01:02:14.451800 2341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:14.441446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39297 (local address 127.0.98.1:49031)
0929 01:02:14.444551 (+ 3105us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:14.444567 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:14.444761 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:14.445157 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:14.445160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:14.445386 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:14.445614 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:14.445621 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.446449 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.446452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:14.447085 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:14.447093 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.447520 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.447524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:14.447618 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:14.448152 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:14.448163 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:14.448473 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:14.450210 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:14.450214 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:14.450217 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:14.450464 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:14.450739 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:14.450741 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:14.450743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:14.450896 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:14.451258 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:14.451261 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:14.451368 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 01:02:14.451467 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 01:02:14.451605 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2996,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 01:02:14.465915 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39297
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:14.574046 2343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:14.574301 2343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:14.574383 2343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:14.578667 2343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:14.578728 2343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:14.578768 2343 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 01:02:14.578840 2343 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:02:14.578917 2343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:02:14.584180 2343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:14.585176 2343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:14.586174 2343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:14.595849 2348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:14.595831 2351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:14.595888 2349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:14.596076 2343 server_base.cc:1034] running on GCE node
I20260929 01:02:14.596487 2343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:14.597033 2343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:14.598183 2343 hybrid_clock.cc:648] HybridClock initialized: now 1790643734598155 us; error 42 us; skew 500 ppm
Sep 29 01:02:14 dist-test-slave-7zdl krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:14.600818 2343 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:02:14.601687 2343 webserver.cc:460] Webserver started at http://127.0.98.2:36931/ using document root <none> and password file <none>
I20260929 01:02:14.602237 2343 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:14.602296 2343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:14.602521 2343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:14.604557 2343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "17fc9c40b7a24115b9d935449d67e1d7"
format_stamp: "Formatted at 2026-09-29 01:02:14 on dist-test-slave-7zdl"
server_key: "332bb57c40d719ea254a7d3c3269aafe"
server_key_iv: "3b91404c24e11f74d03368aa1e0bea94"
server_key_version: "encryptionkey@0"
I20260929 01:02:14.605103 2343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "17fc9c40b7a24115b9d935449d67e1d7"
format_stamp: "Formatted at 2026-09-29 01:02:14 on dist-test-slave-7zdl"
server_key: "332bb57c40d719ea254a7d3c3269aafe"
server_key_iv: "3b91404c24e11f74d03368aa1e0bea94"
server_key_version: "encryptionkey@0"
I20260929 01:02:14.608665 2343 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:02:14.611059 2358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:14.612207 2343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:14.612339 2343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "17fc9c40b7a24115b9d935449d67e1d7"
format_stamp: "Formatted at 2026-09-29 01:02:14 on dist-test-slave-7zdl"
server_key: "332bb57c40d719ea254a7d3c3269aafe"
server_key_iv: "3b91404c24e11f74d03368aa1e0bea94"
server_key_version: "encryptionkey@0"
I20260929 01:02:14.612443 2343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:14.630817 2343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:14.633495 2343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:14.633698 2343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:14.635121 2343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:14.635193 2343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:14.635272 2343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:14.635322 2343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:02:14 dist-test-slave-7zdl krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:14.645938 2343 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34141
I20260929 01:02:14.646021 2473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34141 every 8 connection(s)
I20260929 01:02:14.646906 2343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:02:14.648901 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:14.635851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43451 (local address 127.0.98.62:39297)
0929 01:02:14.636011 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:14.636017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:14.636819 (+ 802us) server_negotiation.cc:407] Connection header received
0929 01:02:14.637817 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:14.637821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:14.637874 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:14.637956 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:14.639301 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.640244 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.641686 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.641914 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.642030 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:14.645168 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:14.645174 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:14.645181 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:14.645240 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:14.646845 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:14.647334 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:14.647339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:14.647341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:14.647429 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:14.647827 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:14.647831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:14.647833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:14.648023 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:14.648131 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:14.648607 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 01:02:14.648732 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 01:02:14.649623 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:14.636217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39297 (local address 127.0.98.2:43451)
0929 01:02:14.636654 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:14.636701 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:14.637611 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:14.638255 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:14.638263 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:14.638567 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:14.639158 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:14.639174 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.640385 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.640391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:14.641557 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:14.641566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.642073 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.642077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:14.642289 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:14.642893 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:14.642909 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:14.644985 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:14.646940 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:14.646946 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:14.646959 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:14.647229 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:14.647527 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:14.647530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:14.647532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:14.647703 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:14.648128 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:14.648134 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:14.648367 (+ 233us) client_negotiation.cc:769] Sending connection context
0929 01:02:14.648554 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 01:02:14.648859 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":132,"threads_started":1}
I20260929 01:02:14.650849 2474 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39297
I20260929 01:02:14.651021 2474 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:14.651327 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 2343
I20260929 01:02:14.651441 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:02:14.651571 2474 heartbeater.cc:511] Master 127.0.98.62:39297 requested a full tablet report, sending...
I20260929 01:02:14.651791 392 external_mini_cluster.cc:1428] Setting key 19019f566afd33c00f605716184380d4
I20260929 01:02:14.653053 2130 ts_manager.cc:194] Registered new tserver with Master: 17fc9c40b7a24115b9d935449d67e1d7 (127.0.98.2:34141)
I20260929 01:02:14.653635 2130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43451
I20260929 01:02:14.661727 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:14.653914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44799 (local address 127.0.98.62:39297)
0929 01:02:14.654045 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:14.654051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:14.654280 (+ 229us) server_negotiation.cc:407] Connection header received
0929 01:02:14.654376 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:14.654382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:14.654435 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:14.654519 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:14.655022 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.655845 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.656652 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.656958 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.657066 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:14.658417 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:14.658420 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:14.658422 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:14.658444 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:14.660033 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:14.660516 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:14.660518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:14.660519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:14.660598 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:14.660898 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:14.660901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:14.660902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:14.661116 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:14.661200 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:14.661526 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 01:02:14.661619 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 01:02:14.661850 2480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:14.652977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39297 (local address 127.0.98.2:44799)
0929 01:02:14.654182 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:14.654197 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:14.654298 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:14.654558 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:14.654560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:14.654763 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:14.654896 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:14.654903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.655975 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.655977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:14.656512 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:14.656520 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.657099 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.657105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:14.657238 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:14.657746 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:14.657765 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:14.658241 (+ 476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:14.660148 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:14.660155 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:14.660158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:14.660415 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:14.660694 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:14.660699 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:14.660701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:14.660799 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:14.661220 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:14.661224 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:14.661331 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 01:02:14.661447 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 01:02:14.661611 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1120,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 01:02:14.681634 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39297
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:14.791221 2483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:14.791486 2483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:14.791560 2483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:14.795508 2483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:14.795583 2483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:14.795648 2483 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 01:02:14.795689 2483 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:02:14.795756 2483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:02:14.800933 2483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:14.801992 2483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:14.802992 2483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:14.812072 2489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:14.812141 2491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:14.812106 2488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:14.812327 2483 server_base.cc:1034] running on GCE node
I20260929 01:02:14.812757 2483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:14.813398 2483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:14.814607 2483 hybrid_clock.cc:648] HybridClock initialized: now 1790643734814580 us; error 46 us; skew 500 ppm
Sep 29 01:02:14 dist-test-slave-7zdl krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:14.817310 2483 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:02:14.818123 2483 webserver.cc:460] Webserver started at http://127.0.98.3:46811/ using document root <none> and password file <none>
I20260929 01:02:14.818681 2483 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:14.818769 2483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:14.819038 2483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:14.820976 2483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "39cdc4e943be4c90a96b54aa8654ca0c"
format_stamp: "Formatted at 2026-09-29 01:02:14 on dist-test-slave-7zdl"
server_key: "943e9250139136c4051e0a7609916ef0"
server_key_iv: "fdf5fbd9d64917e1dd8af58dd467f547"
server_key_version: "encryptionkey@0"
I20260929 01:02:14.821533 2483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "39cdc4e943be4c90a96b54aa8654ca0c"
format_stamp: "Formatted at 2026-09-29 01:02:14 on dist-test-slave-7zdl"
server_key: "943e9250139136c4051e0a7609916ef0"
server_key_iv: "fdf5fbd9d64917e1dd8af58dd467f547"
server_key_version: "encryptionkey@0"
I20260929 01:02:14.825505 2483 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 01:02:14.827937 2498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:14.829142 2483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:02:14.829285 2483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "39cdc4e943be4c90a96b54aa8654ca0c"
format_stamp: "Formatted at 2026-09-29 01:02:14 on dist-test-slave-7zdl"
server_key: "943e9250139136c4051e0a7609916ef0"
server_key_iv: "fdf5fbd9d64917e1dd8af58dd467f547"
server_key_version: "encryptionkey@0"
I20260929 01:02:14.829413 2483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:14.843891 2483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:14.847118 2483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:14.847342 2483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:14.848834 2483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:14.848886 2483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:14.848958 2483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:14.848996 2483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:02:14 dist-test-slave-7zdl krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:14.860031 2483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41805
I20260929 01:02:14.860081 2613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41805 every 8 connection(s)
I20260929 01:02:14.861115 2483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:02:14.863583 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:14.849655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40471 (local address 127.0.98.62:39297)
0929 01:02:14.849796 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:14.849801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:14.850854 (+ 1053us) server_negotiation.cc:407] Connection header received
0929 01:02:14.852078 (+ 1224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:14.852085 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:14.852157 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:14.852277 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:14.853849 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.854882 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.856781 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.857114 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.857238 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:14.860077 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:14.860081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:14.860084 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:14.860115 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:14.861692 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:14.862247 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:14.862250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:14.862251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:14.862329 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:14.862596 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:14.862599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:14.862600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:14.862765 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:14.862919 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:14.863336 (+ 417us) server_negotiation.cc:299] Negotiation successful
0929 01:02:14.863435 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 01:02:14.864401 2507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:14.850168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39297 (local address 127.0.98.3:40471)
0929 01:02:14.850662 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:14.850709 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:14.851791 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:14.852467 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:14.852477 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:14.852970 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:14.853657 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:14.853681 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.855023 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.855028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:14.856526 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:14.856565 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.857283 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.857287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:14.857469 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:14.858099 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:14.858114 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:14.859894 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:14.861814 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:14.861819 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:14.861832 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:14.862157 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:14.862406 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:14.862408 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:14.862410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:14.862522 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:14.862915 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:14.862920 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:14.863134 (+ 214us) client_negotiation.cc:769] Sending connection context
0929 01:02:14.863284 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 01:02:14.863565 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":143,"threads_started":1}
I20260929 01:02:14.865674 2614 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39297
I20260929 01:02:14.865824 2614 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:14.866322 2614 heartbeater.cc:511] Master 127.0.98.62:39297 requested a full tablet report, sending...
I20260929 01:02:14.867357 2131 ts_manager.cc:194] Registered new tserver with Master: 39cdc4e943be4c90a96b54aa8654ca0c (127.0.98.3:41805)
I20260929 01:02:14.867363 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 2483
I20260929 01:02:14.867466 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:02:14.867919 392 external_mini_cluster.cc:1428] Setting key be14b87a39bb1cee2f34205c23bb44da
I20260929 01:02:14.868146 2131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40471
I20260929 01:02:14.869849 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:02:14.876048 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:14.867904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35005 (local address 127.0.98.62:39297)
0929 01:02:14.868012 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:14.868016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:14.868720 (+ 704us) server_negotiation.cc:407] Connection header received
0929 01:02:14.868869 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:14.868872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:14.868940 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:14.869038 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:14.869482 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.870199 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.871138 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.871542 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.871655 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:14.872923 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:14.872926 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:14.872928 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:14.872947 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:14.874444 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:14.874894 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:14.874897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:14.874899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:14.874980 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:14.875267 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:14.875272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:14.875273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:14.875435 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:14.875544 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:14.875800 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 01:02:14.875901 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 01:02:14.876077 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:14.867829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39297 (local address 127.0.98.3:35005)
0929 01:02:14.868644 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:14.868666 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:14.868781 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:14.869058 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:14.869061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:14.869264 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:14.869366 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:14.869373 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.870335 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.870338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:14.871028 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:14.871037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.871690 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.871695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:14.871814 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:14.872287 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:14.872304 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:14.872791 (+ 487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:14.874560 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:14.874565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:14.874567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:14.874762 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:14.875090 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:14.875093 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:14.875095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:14.875178 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:14.875536 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:14.875540 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:14.875633 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 01:02:14.875736 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 01:02:14.875871 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":108,"threads_started":1}
Sep 29 01:02:14 dist-test-slave-7zdl krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 01:02:14 dist-test-slave-7zdl krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:14.895020 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:14.887344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47696 (local address 127.0.98.62:39297)
0929 01:02:14.887495 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:14.887504 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:14.887516 (+ 12us) server_negotiation.cc:407] Connection header received
0929 01:02:14.887559 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:14.887561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:14.887622 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:14.887696 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:14.888436 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.889089 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.889880 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.890084 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.890159 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:14.892465 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:14.892467 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:14.892469 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:14.892490 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:14.893686 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:14.894053 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:14.894056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:14.894057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:14.894124 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:14.894331 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:14.894333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:14.894334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:14.894485 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:14.894588 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:14.894754 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 01:02:14.894903 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:02:14.897656 2131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:02:14.900789 2131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:02:14.914706 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:14.908711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45525 (local address 127.0.0.1:46686)
0929 01:02:14.909412 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:14.909459 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:14.909600 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:14.910234 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:14.910240 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:14.910266 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:14.910393 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:14.910400 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.911637 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.911644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:14.913086 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:14.913095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.914126 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.914129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:14.914229 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:14.914275 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 01:02:14.914389 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 01:02:14.914503 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":29,"thread_start_us":70,"threads_started":1}
I20260929 01:02:14.915004 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:14.909724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34141 (local address 127.0.0.1:55772)
0929 01:02:14.910079 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:14.910091 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:14.910172 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:14.910909 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:14.910912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:14.910924 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:14.911139 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:14.911150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.912472 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.912475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:14.913554 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:14.913561 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.914424 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.914427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:14.914530 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:14.914569 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 01:02:14.914640 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 01:02:14.914711 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":113,"thread_start_us":75,"threads_started":1}
I20260929 01:02:14.915012 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:14.909427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41805 (local address 127.0.0.1:46748)
0929 01:02:14.909812 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:14.909824 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:14.909896 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:14.910873 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:14.910876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:14.910898 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:14.911084 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:14.911091 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.912493 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.912497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:14.913547 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:14.913559 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.914586 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.914592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:14.914686 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:14.914704 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 01:02:14.914748 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 01:02:14.914824 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":127,"thread_start_us":92,"threads_started":1}
I20260929 01:02:14.915297 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:14.910103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55772 (local address 127.0.98.2:34141)
0929 01:02:14.910517 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:14.910528 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:14.910550 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:02:14.910605 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:14.910608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:14.910745 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:14.911072 (+ 327us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:14.911243 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.912358 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.913665 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.914304 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.914413 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:14.914556 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:14.914678 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 01:02:14.914786 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":112,"threads_started":1}
I20260929 01:02:14.915714 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:14.909594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46748 (local address 127.0.98.3:41805)
0929 01:02:14.910400 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:14.910417 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:14.910433 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:02:14.910519 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:14.910524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:14.910709 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:14.910894 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:14.911216 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.912364 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.913685 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.914452 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.915114 (+ 662us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:14.915416 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:14.915490 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 01:02:14.915581 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":98,"threads_started":1}
I20260929 01:02:14.915814 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:14.908841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46686 (local address 127.0.98.1:45525)
0929 01:02:14.909242 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:14.909258 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:14.909750 (+ 492us) server_negotiation.cc:407] Connection header received
0929 01:02:14.909835 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:14.909842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:14.910064 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:14.910226 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:14.910515 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.911536 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.913213 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:14.914001 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:14.914110 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:14.914268 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:14.915580 (+ 1312us) server_negotiation.cc:299] Negotiation successful
0929 01:02:14.915667 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":84,"threads_started":1}
I20260929 01:02:14.918030 2408 tablet_service.cc:1467] Processing CreateTablet for tablet 12406677897a4430ae5f3c94f8488b4b (DEFAULT_TABLE table=test-table [id=1fa18920de89472f9c16500560b0c048]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:14.918239 2548 tablet_service.cc:1467] Processing CreateTablet for tablet 12406677897a4430ae5f3c94f8488b4b (DEFAULT_TABLE table=test-table [id=1fa18920de89472f9c16500560b0c048]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:14.918239 2268 tablet_service.cc:1467] Processing CreateTablet for tablet 12406677897a4430ae5f3c94f8488b4b (DEFAULT_TABLE table=test-table [id=1fa18920de89472f9c16500560b0c048]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:14.919157 2408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12406677897a4430ae5f3c94f8488b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:14.919250 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12406677897a4430ae5f3c94f8488b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:14.919243 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12406677897a4430ae5f3c94f8488b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:14.925544 2637 tablet_bootstrap.cc:492] T 12406677897a4430ae5f3c94f8488b4b P 17fc9c40b7a24115b9d935449d67e1d7: Bootstrap starting.
I20260929 01:02:14.926937 2638 tablet_bootstrap.cc:492] T 12406677897a4430ae5f3c94f8488b4b P 2ac54ced6dad46969c9b046507046048: Bootstrap starting.
I20260929 01:02:14.927273 2639 tablet_bootstrap.cc:492] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c: Bootstrap starting.
I20260929 01:02:14.927888 2637 tablet_bootstrap.cc:654] T 12406677897a4430ae5f3c94f8488b4b P 17fc9c40b7a24115b9d935449d67e1d7: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:14.928582 2637 log.cc:824] T 12406677897a4430ae5f3c94f8488b4b P 17fc9c40b7a24115b9d935449d67e1d7: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:14.928942 2638 tablet_bootstrap.cc:654] T 12406677897a4430ae5f3c94f8488b4b P 2ac54ced6dad46969c9b046507046048: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:14.928942 2639 tablet_bootstrap.cc:654] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:14.929746 2639 log.cc:824] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:14.929746 2638 log.cc:824] T 12406677897a4430ae5f3c94f8488b4b P 2ac54ced6dad46969c9b046507046048: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:14.930617 2637 tablet_bootstrap.cc:492] T 12406677897a4430ae5f3c94f8488b4b P 17fc9c40b7a24115b9d935449d67e1d7: No bootstrap required, opened a new log
I20260929 01:02:14.930821 2637 ts_tablet_manager.cc:1397] T 12406677897a4430ae5f3c94f8488b4b P 17fc9c40b7a24115b9d935449d67e1d7: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 01:02:14.931595 2638 tablet_bootstrap.cc:492] T 12406677897a4430ae5f3c94f8488b4b P 2ac54ced6dad46969c9b046507046048: No bootstrap required, opened a new log
I20260929 01:02:14.931774 2638 ts_tablet_manager.cc:1397] T 12406677897a4430ae5f3c94f8488b4b P 2ac54ced6dad46969c9b046507046048: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 01:02:14.931967 2639 tablet_bootstrap.cc:492] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c: No bootstrap required, opened a new log
I20260929 01:02:14.932118 2639 ts_tablet_manager.cc:1397] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 01:02:14.934762 2637 raft_consensus.cc:348] T 12406677897a4430ae5f3c94f8488b4b P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } }
I20260929 01:02:14.935051 2637 raft_consensus.cc:374] T 12406677897a4430ae5f3c94f8488b4b P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:14.935105 2637 raft_consensus.cc:729] T 12406677897a4430ae5f3c94f8488b4b P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17fc9c40b7a24115b9d935449d67e1d7, State: Initialized, Role: FOLLOWER
I20260929 01:02:14.935562 2637 consensus_queue.cc:260] T 12406677897a4430ae5f3c94f8488b4b P 17fc9c40b7a24115b9d935449d67e1d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } }
I20260929 01:02:14.935518 2638 raft_consensus.cc:348] T 12406677897a4430ae5f3c94f8488b4b P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } }
I20260929 01:02:14.935732 2638 raft_consensus.cc:374] T 12406677897a4430ae5f3c94f8488b4b P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:14.935777 2638 raft_consensus.cc:729] T 12406677897a4430ae5f3c94f8488b4b P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac54ced6dad46969c9b046507046048, State: Initialized, Role: FOLLOWER
I20260929 01:02:14.935832 2639 raft_consensus.cc:348] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } }
I20260929 01:02:14.936043 2639 raft_consensus.cc:374] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:14.936087 2639 raft_consensus.cc:729] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39cdc4e943be4c90a96b54aa8654ca0c, State: Initialized, Role: FOLLOWER
I20260929 01:02:14.936193 2638 consensus_queue.cc:260] T 12406677897a4430ae5f3c94f8488b4b P 2ac54ced6dad46969c9b046507046048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } }
I20260929 01:02:14.936573 2637 ts_tablet_manager.cc:1428] T 12406677897a4430ae5f3c94f8488b4b P 17fc9c40b7a24115b9d935449d67e1d7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 01:02:14.936564 2639 consensus_queue.cc:260] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } }
I20260929 01:02:14.936896 2334 heartbeater.cc:503] Master 127.0.98.62:39297 was elected leader, sending a full tablet report...
I20260929 01:02:14.937067 2638 ts_tablet_manager.cc:1428] T 12406677897a4430ae5f3c94f8488b4b P 2ac54ced6dad46969c9b046507046048: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260929 01:02:14.937170 2614 heartbeater.cc:503] Master 127.0.98.62:39297 was elected leader, sending a full tablet report...
I20260929 01:02:14.937397 2639 ts_tablet_manager.cc:1428] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 01:02:14.937901 2474 heartbeater.cc:503] Master 127.0.98.62:39297 was elected leader, sending a full tablet report...
I20260929 01:02:15.009779 2645 raft_consensus.cc:482] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:02:15.009951 2645 raft_consensus.cc:504] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } }
I20260929 01:02:15.011008 2645 leader_election.cc:288] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ac54ced6dad46969c9b046507046048 (127.0.98.1:45525), 17fc9c40b7a24115b9d935449d67e1d7 (127.0.98.2:34141)
I20260929 01:02:15.015249 2507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.011276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45525 (local address 127.0.98.3:50631)
0929 01:02:15.011420 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:15.011442 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:15.011555 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:15.011859 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:15.011863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:15.011882 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:15.012023 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:15.012031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.012881 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.012885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:15.013794 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:15.013803 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.014635 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.014671 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:15.014843 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.014879 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 01:02:15.014981 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 01:02:15.015077 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 01:02:15.015311 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.011353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50631 (local address 127.0.98.1:45525)
0929 01:02:15.011505 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:15.011510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:15.011526 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:02:15.011687 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:15.011690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:15.011765 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:15.011844 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:15.012142 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.012778 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.013915 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.014497 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.014615 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.014695 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:15.015008 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 01:02:15.015136 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 01:02:15.015892 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12406677897a4430ae5f3c94f8488b4b" candidate_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac54ced6dad46969c9b046507046048" is_pre_election: true
I20260929 01:02:15.016127 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.011550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34141 (local address 127.0.98.3:41235)
0929 01:02:15.011894 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:15.011907 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:15.011996 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:15.012246 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:15.012249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:15.012265 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:15.012378 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:15.012384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.013185 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.013188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:15.014616 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:15.014625 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.015713 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.015717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:15.015826 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.015847 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 01:02:15.015925 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 01:02:15.015988 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":110,"threads_started":1}
I20260929 01:02:15.016148 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.011713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41235 (local address 127.0.98.2:34141)
0929 01:02:15.011843 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:15.011847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:15.011993 (+ 146us) server_negotiation.cc:407] Connection header received
0929 01:02:15.012094 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:15.012100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:15.012156 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:15.012210 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:15.012482 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.013078 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.014746 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.015432 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.015516 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.015572 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:15.015940 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 01:02:15.016027 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 01:02:15.016201 2288 raft_consensus.cc:2454] T 12406677897a4430ae5f3c94f8488b4b P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39cdc4e943be4c90a96b54aa8654ca0c in term 0.
I20260929 01:02:15.016696 2499 leader_election.cc:302] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ac54ced6dad46969c9b046507046048, 39cdc4e943be4c90a96b54aa8654ca0c; no voters:
I20260929 01:02:15.016713 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12406677897a4430ae5f3c94f8488b4b" candidate_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17fc9c40b7a24115b9d935449d67e1d7" is_pre_election: true
I20260929 01:02:15.016979 2426 raft_consensus.cc:2454] T 12406677897a4430ae5f3c94f8488b4b P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39cdc4e943be4c90a96b54aa8654ca0c in term 0.
I20260929 01:02:15.016999 2645 raft_consensus.cc:2781] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:02:15.017129 2645 raft_consensus.cc:482] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:02:15.017187 2645 raft_consensus.cc:3037] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:15.018263 2645 raft_consensus.cc:504] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } }
I20260929 01:02:15.018644 2645 leader_election.cc:288] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c [CANDIDATE]: Term 1 election: Requested vote from peers 2ac54ced6dad46969c9b046507046048 (127.0.98.1:45525), 17fc9c40b7a24115b9d935449d67e1d7 (127.0.98.2:34141)
I20260929 01:02:15.018944 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12406677897a4430ae5f3c94f8488b4b" candidate_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac54ced6dad46969c9b046507046048"
I20260929 01:02:15.018990 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12406677897a4430ae5f3c94f8488b4b" candidate_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17fc9c40b7a24115b9d935449d67e1d7"
I20260929 01:02:15.019062 2288 raft_consensus.cc:3037] T 12406677897a4430ae5f3c94f8488b4b P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:15.019116 2426 raft_consensus.cc:3037] T 12406677897a4430ae5f3c94f8488b4b P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:15.020023 2288 raft_consensus.cc:2454] T 12406677897a4430ae5f3c94f8488b4b P 2ac54ced6dad46969c9b046507046048 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39cdc4e943be4c90a96b54aa8654ca0c in term 1.
I20260929 01:02:15.020300 2499 leader_election.cc:302] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ac54ced6dad46969c9b046507046048, 39cdc4e943be4c90a96b54aa8654ca0c; no voters:
I20260929 01:02:15.020463 2645 raft_consensus.cc:2781] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:15.020612 2426 raft_consensus.cc:2454] T 12406677897a4430ae5f3c94f8488b4b P 17fc9c40b7a24115b9d935449d67e1d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39cdc4e943be4c90a96b54aa8654ca0c in term 1.
I20260929 01:02:15.020704 2645 raft_consensus.cc:686] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c [term 1 LEADER]: Becoming Leader. State: Replica: 39cdc4e943be4c90a96b54aa8654ca0c, State: Running, Role: LEADER
I20260929 01:02:15.021080 2645 consensus_queue.cc:237] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } }
I20260929 01:02:15.024171 2131 catalog_manager.cc:5495] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 39cdc4e943be4c90a96b54aa8654ca0c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } health_report { overall_health: UNKNOWN } } }
Sep 29 01:02:15 dist-test-slave-7zdl krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:15.089057 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.080519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47710 (local address 127.0.98.62:39297)
0929 01:02:15.080680 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:15.080685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:15.080824 (+ 139us) server_negotiation.cc:407] Connection header received
0929 01:02:15.080990 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:15.081003 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:15.081037 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:15.081119 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:15.081794 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.082311 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.083219 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.083434 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.083543 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.085843 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:15.085846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:15.085848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:15.085869 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:15.087343 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:15.087866 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:15.087868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:15.087870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:15.087939 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:15.088213 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:15.088215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:15.088216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:15.088380 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:15.088451 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:15.088725 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 01:02:15.088934 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":69}
I20260929 01:02:15.089072 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.080393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39297 (local address 127.0.0.1:47710)
0929 01:02:15.080735 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:15.080748 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:15.080888 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:15.081144 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:15.081147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:15.081590 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:15.081677 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:15.081683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.082434 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.082437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:15.083107 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:15.083117 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.083542 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.083545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:15.083637 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.084198 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:15.084210 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:15.085705 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:15.087438 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:15.087445 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:15.087466 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:15.087721 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:15.088059 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:15.088061 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:15.088062 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:15.088138 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:15.088461 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:15.088466 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:15.088579 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 01:02:15.088675 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 01:02:15.088864 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
I20260929 01:02:15.092895 2131 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47710:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 01:02:15.093603 2131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:02:15.098104 2408 tablet_service.cc:1467] Processing CreateTablet for tablet 9611cb2609e548f7b69e15ff017d2e9d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e90de56b5f9246c5992979f6129ea83d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:02:15.098119 2548 tablet_service.cc:1467] Processing CreateTablet for tablet 9611cb2609e548f7b69e15ff017d2e9d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e90de56b5f9246c5992979f6129ea83d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:02:15.098124 2268 tablet_service.cc:1467] Processing CreateTablet for tablet 9611cb2609e548f7b69e15ff017d2e9d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e90de56b5f9246c5992979f6129ea83d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:02:15.098870 2408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9611cb2609e548f7b69e15ff017d2e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:15.098871 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9611cb2609e548f7b69e15ff017d2e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:15.098891 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9611cb2609e548f7b69e15ff017d2e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:15.101549 2637 tablet_bootstrap.cc:492] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7: Bootstrap starting.
I20260929 01:02:15.101989 2639 tablet_bootstrap.cc:492] T 9611cb2609e548f7b69e15ff017d2e9d P 39cdc4e943be4c90a96b54aa8654ca0c: Bootstrap starting.
I20260929 01:02:15.102547 2637 tablet_bootstrap.cc:654] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:15.102866 2639 tablet_bootstrap.cc:654] T 9611cb2609e548f7b69e15ff017d2e9d P 39cdc4e943be4c90a96b54aa8654ca0c: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:15.103616 2638 tablet_bootstrap.cc:492] T 9611cb2609e548f7b69e15ff017d2e9d P 2ac54ced6dad46969c9b046507046048: Bootstrap starting.
I20260929 01:02:15.104101 2639 tablet_bootstrap.cc:492] T 9611cb2609e548f7b69e15ff017d2e9d P 39cdc4e943be4c90a96b54aa8654ca0c: No bootstrap required, opened a new log
I20260929 01:02:15.104157 2637 tablet_bootstrap.cc:492] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7: No bootstrap required, opened a new log
I20260929 01:02:15.104182 2639 ts_tablet_manager.cc:1397] T 9611cb2609e548f7b69e15ff017d2e9d P 39cdc4e943be4c90a96b54aa8654ca0c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 01:02:15.104233 2637 ts_tablet_manager.cc:1397] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 01:02:15.104586 2638 tablet_bootstrap.cc:654] T 9611cb2609e548f7b69e15ff017d2e9d P 2ac54ced6dad46969c9b046507046048: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:15.104668 2639 raft_consensus.cc:348] T 9611cb2609e548f7b69e15ff017d2e9d P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:15.104754 2639 raft_consensus.cc:374] T 9611cb2609e548f7b69e15ff017d2e9d P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:15.104696 2637 raft_consensus.cc:348] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:15.104775 2639 raft_consensus.cc:729] T 9611cb2609e548f7b69e15ff017d2e9d P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39cdc4e943be4c90a96b54aa8654ca0c, State: Initialized, Role: FOLLOWER
I20260929 01:02:15.104802 2637 raft_consensus.cc:374] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:15.104851 2637 raft_consensus.cc:729] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17fc9c40b7a24115b9d935449d67e1d7, State: Initialized, Role: FOLLOWER
I20260929 01:02:15.104898 2639 consensus_queue.cc:260] T 9611cb2609e548f7b69e15ff017d2e9d P 39cdc4e943be4c90a96b54aa8654ca0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:15.104983 2637 consensus_queue.cc:260] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:15.105188 2639 ts_tablet_manager.cc:1428] T 9611cb2609e548f7b69e15ff017d2e9d P 39cdc4e943be4c90a96b54aa8654ca0c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 01:02:15.105159 2645 tablet_replica.cc:439] T 9611cb2609e548f7b69e15ff017d2e9d P 39cdc4e943be4c90a96b54aa8654ca0c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } } }
I20260929 01:02:15.105275 2637 ts_tablet_manager.cc:1428] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 01:02:15.105315 2645 tablet_replica.cc:442] T 9611cb2609e548f7b69e15ff017d2e9d P 39cdc4e943be4c90a96b54aa8654ca0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:02:15.105548 2643 tablet_replica.cc:439] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } } }
I20260929 01:02:15.105669 2643 tablet_replica.cc:442] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:02:15.106935 2638 tablet_bootstrap.cc:492] T 9611cb2609e548f7b69e15ff017d2e9d P 2ac54ced6dad46969c9b046507046048: No bootstrap required, opened a new log
I20260929 01:02:15.107048 2638 ts_tablet_manager.cc:1397] T 9611cb2609e548f7b69e15ff017d2e9d P 2ac54ced6dad46969c9b046507046048: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 01:02:15.107499 2638 raft_consensus.cc:348] T 9611cb2609e548f7b69e15ff017d2e9d P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:15.107630 2638 raft_consensus.cc:374] T 9611cb2609e548f7b69e15ff017d2e9d P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:15.107671 2638 raft_consensus.cc:729] T 9611cb2609e548f7b69e15ff017d2e9d P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac54ced6dad46969c9b046507046048, State: Initialized, Role: FOLLOWER
I20260929 01:02:15.107797 2638 consensus_queue.cc:260] T 9611cb2609e548f7b69e15ff017d2e9d P 2ac54ced6dad46969c9b046507046048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:15.108103 2638 ts_tablet_manager.cc:1428] T 9611cb2609e548f7b69e15ff017d2e9d P 2ac54ced6dad46969c9b046507046048: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:15.108064 2644 tablet_replica.cc:439] T 9611cb2609e548f7b69e15ff017d2e9d P 2ac54ced6dad46969c9b046507046048: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } } }
I20260929 01:02:15.108203 2644 tablet_replica.cc:442] T 9611cb2609e548f7b69e15ff017d2e9d P 2ac54ced6dad46969c9b046507046048: This TxnStatusTablet replica's current role is: FOLLOWER
W20260929 01:02:15.112237 2615 tablet.cc:2369] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:02:15.148202 2475 tablet.cc:2369] T 12406677897a4430ae5f3c94f8488b4b P 17fc9c40b7a24115b9d935449d67e1d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:02:15.186600 2335 tablet.cc:2369] T 12406677897a4430ae5f3c94f8488b4b P 2ac54ced6dad46969c9b046507046048: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:02:15.196648 2643 raft_consensus.cc:482] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:02:15.196797 2643 raft_consensus.cc:504] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:15.198251 2643 leader_election.cc:288] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ac54ced6dad46969c9b046507046048 (127.0.98.1:45525), 39cdc4e943be4c90a96b54aa8654ca0c (127.0.98.3:41805)
I20260929 01:02:15.202288 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.198438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45525 (local address 127.0.98.2:34999)
0929 01:02:15.198679 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:15.198692 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:15.198827 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:15.199157 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:15.199163 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:15.199185 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:15.199321 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:15.199330 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.200174 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.200177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:15.201202 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:15.201211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.201853 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.201857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:15.201966 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.201999 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 01:02:15.202064 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 01:02:15.202142 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":66,"threads_started":1}
I20260929 01:02:15.202312 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.198578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34999 (local address 127.0.98.1:45525)
0929 01:02:15.198743 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:15.198748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:15.198766 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:02:15.198964 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:15.198968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:15.199019 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:15.199093 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:15.199433 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.200073 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.201321 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.201746 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.201824 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.201874 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:15.202102 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 01:02:15.202177 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 01:02:15.202489 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.198926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57925 (local address 127.0.98.3:41805)
0929 01:02:15.199056 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:15.199060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:15.199135 (+ 75us) server_negotiation.cc:407] Connection header received
0929 01:02:15.199228 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:15.199231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:15.199279 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:15.199363 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:15.199618 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.200257 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.201389 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.201888 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.202016 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.202087 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:15.202284 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 01:02:15.202364 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 01:02:15.202508 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.198778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41805 (local address 127.0.98.2:57925)
0929 01:02:15.199047 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:15.199061 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:15.199140 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:15.199373 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:15.199375 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:15.199411 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:15.199514 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:15.199520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.200414 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.200418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:15.201220 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:15.201228 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.202025 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.202029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:15.202157 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.202179 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 01:02:15.202253 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 01:02:15.202312 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":3,"thread_start_us":80,"threads_started":1}
I20260929 01:02:15.202729 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9611cb2609e548f7b69e15ff017d2e9d" candidate_uuid: "17fc9c40b7a24115b9d935449d67e1d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac54ced6dad46969c9b046507046048" is_pre_election: true
I20260929 01:02:15.202914 2288 raft_consensus.cc:2454] T 9611cb2609e548f7b69e15ff017d2e9d P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17fc9c40b7a24115b9d935449d67e1d7 in term 0.
I20260929 01:02:15.203193 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9611cb2609e548f7b69e15ff017d2e9d" candidate_uuid: "17fc9c40b7a24115b9d935449d67e1d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" is_pre_election: true
I20260929 01:02:15.203289 2359 leader_election.cc:302] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17fc9c40b7a24115b9d935449d67e1d7, 2ac54ced6dad46969c9b046507046048; no voters:
I20260929 01:02:15.203397 2568 raft_consensus.cc:2454] T 9611cb2609e548f7b69e15ff017d2e9d P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17fc9c40b7a24115b9d935449d67e1d7 in term 0.
I20260929 01:02:15.203581 2643 raft_consensus.cc:2781] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:02:15.203666 2643 raft_consensus.cc:482] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:02:15.203707 2643 raft_consensus.cc:3037] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:15.204675 2643 raft_consensus.cc:504] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:15.204991 2643 leader_election.cc:288] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7 [CANDIDATE]: Term 1 election: Requested vote from peers 2ac54ced6dad46969c9b046507046048 (127.0.98.1:45525), 39cdc4e943be4c90a96b54aa8654ca0c (127.0.98.3:41805)
I20260929 01:02:15.205319 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9611cb2609e548f7b69e15ff017d2e9d" candidate_uuid: "17fc9c40b7a24115b9d935449d67e1d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac54ced6dad46969c9b046507046048"
I20260929 01:02:15.205368 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9611cb2609e548f7b69e15ff017d2e9d" candidate_uuid: "17fc9c40b7a24115b9d935449d67e1d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39cdc4e943be4c90a96b54aa8654ca0c"
I20260929 01:02:15.205446 2288 raft_consensus.cc:3037] T 9611cb2609e548f7b69e15ff017d2e9d P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:15.205518 2568 raft_consensus.cc:3037] T 9611cb2609e548f7b69e15ff017d2e9d P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:15.206351 2288 raft_consensus.cc:2454] T 9611cb2609e548f7b69e15ff017d2e9d P 2ac54ced6dad46969c9b046507046048 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17fc9c40b7a24115b9d935449d67e1d7 in term 1.
I20260929 01:02:15.206425 2568 raft_consensus.cc:2454] T 9611cb2609e548f7b69e15ff017d2e9d P 39cdc4e943be4c90a96b54aa8654ca0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17fc9c40b7a24115b9d935449d67e1d7 in term 1.
I20260929 01:02:15.206686 2359 leader_election.cc:302] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17fc9c40b7a24115b9d935449d67e1d7, 2ac54ced6dad46969c9b046507046048; no voters:
I20260929 01:02:15.206880 2643 raft_consensus.cc:2781] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:15.207073 2643 raft_consensus.cc:686] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7 [term 1 LEADER]: Becoming Leader. State: Replica: 17fc9c40b7a24115b9d935449d67e1d7, State: Running, Role: LEADER
I20260929 01:02:15.207404 2643 consensus_queue.cc:237] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:15.209404 2660 tablet_replica.cc:439] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7: TxnStatusTablet state changed. Reason: New leader 17fc9c40b7a24115b9d935449d67e1d7. Latest consensus state: current_term: 1 leader_uuid: "17fc9c40b7a24115b9d935449d67e1d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } } }
I20260929 01:02:15.209532 2660 tablet_replica.cc:442] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:02:15.209801 2662 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 01:02:15.210027 2662 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 01:02:15.210566 2130 catalog_manager.cc:5495] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17fc9c40b7a24115b9d935449d67e1d7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "17fc9c40b7a24115b9d935449d67e1d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } health_report { overall_health: UNKNOWN } } }
I20260929 01:02:15.231189 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.227148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34141 (local address 127.0.0.1:55778)
0929 01:02:15.227394 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:15.227408 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:15.227509 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:15.227840 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:15.227843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:15.227850 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:02:15.227946 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:15.227950 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.228923 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.228925 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:15.229544 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:15.229553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.230096 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.230099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:15.230166 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.230205 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:02:15.230935 (+ 730us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:02:15.230948 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 01:02:15.231004 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 01:02:15.231077 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 01:02:15.231205 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.227258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55778 (local address 127.0.98.2:34141)
0929 01:02:15.227375 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:15.227380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:15.227473 (+ 93us) server_negotiation.cc:407] Connection header received
0929 01:02:15.227637 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:15.227640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:15.227703 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:15.227806 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:02:15.228045 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.228812 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.229651 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.229995 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.230095 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.230332 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:02:15.230839 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:02:15.230946 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:15.231028 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 01:02:15.231079 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 01:02:15.236618 2568 raft_consensus.cc:1261] T 9611cb2609e548f7b69e15ff017d2e9d P 39cdc4e943be4c90a96b54aa8654ca0c [term 1 FOLLOWER]: Refusing update from remote peer 17fc9c40b7a24115b9d935449d67e1d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:15.236691 2288 raft_consensus.cc:1261] T 9611cb2609e548f7b69e15ff017d2e9d P 2ac54ced6dad46969c9b046507046048 [term 1 FOLLOWER]: Refusing update from remote peer 17fc9c40b7a24115b9d935449d67e1d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:15.237303 2660 consensus_queue.cc:1035] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:15.237576 2643 consensus_queue.cc:1035] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:15.240949 2645 tablet_replica.cc:439] T 9611cb2609e548f7b69e15ff017d2e9d P 39cdc4e943be4c90a96b54aa8654ca0c: TxnStatusTablet state changed. Reason: New leader 17fc9c40b7a24115b9d935449d67e1d7. Latest consensus state: current_term: 1 leader_uuid: "17fc9c40b7a24115b9d935449d67e1d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } } }
I20260929 01:02:15.241080 2645 tablet_replica.cc:442] T 9611cb2609e548f7b69e15ff017d2e9d P 39cdc4e943be4c90a96b54aa8654ca0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:02:15.241163 2644 tablet_replica.cc:439] T 9611cb2609e548f7b69e15ff017d2e9d P 2ac54ced6dad46969c9b046507046048: TxnStatusTablet state changed. Reason: New leader 17fc9c40b7a24115b9d935449d67e1d7. Latest consensus state: current_term: 1 leader_uuid: "17fc9c40b7a24115b9d935449d67e1d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } } }
I20260929 01:02:15.241274 2644 tablet_replica.cc:442] T 9611cb2609e548f7b69e15ff017d2e9d P 2ac54ced6dad46969c9b046507046048: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:02:15.245185 2643 tablet_replica.cc:439] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "17fc9c40b7a24115b9d935449d67e1d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } } }
I20260929 01:02:15.245219 2670 tablet_replica.cc:439] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "17fc9c40b7a24115b9d935449d67e1d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } } }
I20260929 01:02:15.245321 2643 tablet_replica.cc:442] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:02:15.245586 2644 tablet_replica.cc:439] T 9611cb2609e548f7b69e15ff017d2e9d P 2ac54ced6dad46969c9b046507046048: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "17fc9c40b7a24115b9d935449d67e1d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } } }
I20260929 01:02:15.245687 2644 tablet_replica.cc:442] T 9611cb2609e548f7b69e15ff017d2e9d P 2ac54ced6dad46969c9b046507046048: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:02:15.245329 2670 tablet_replica.cc:442] T 9611cb2609e548f7b69e15ff017d2e9d P 17fc9c40b7a24115b9d935449d67e1d7: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:02:15.248793 2645 tablet_replica.cc:439] T 9611cb2609e548f7b69e15ff017d2e9d P 39cdc4e943be4c90a96b54aa8654ca0c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "17fc9c40b7a24115b9d935449d67e1d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } } }
I20260929 01:02:15.248912 2645 tablet_replica.cc:442] T 9611cb2609e548f7b69e15ff017d2e9d P 39cdc4e943be4c90a96b54aa8654ca0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:02:15.250361 2665 mvcc.cc:205] Tried to move back new op lower bound from 7334476739522609152 to 7334476739411914752. Current Snapshot: MvccSnapshot[applied={T|T < 7334476739522609152 or (T in {7334476739522609152})}]
I20260929 01:02:15.259399 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.255566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46760 (local address 127.0.98.3:41805)
0929 01:02:15.255681 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:15.255684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:15.255699 (+ 15us) server_negotiation.cc:407] Connection header received
0929 01:02:15.255802 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:15.255805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:15.255857 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:15.255981 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:02:15.256249 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.256801 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.257786 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.258057 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.258174 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.258469 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:02:15.258990 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:02:15.259112 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:15.259191 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 01:02:15.259246 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 01:02:15.269902 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.266661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34141 (local address 127.0.98.3:35169)
0929 01:02:15.266780 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:15.266832 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:15.266910 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:15.267124 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:15.267126 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:15.267132 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:02:15.267213 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:15.267217 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.268170 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.268172 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:15.268735 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:15.268742 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.269303 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.269306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:15.269371 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.269407 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:02:15.269699 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:02:15.269703 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 01:02:15.269751 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 01:02:15.269808 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 01:02:15.269948 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.266756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35169 (local address 127.0.98.2:34141)
0929 01:02:15.266907 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:15.266911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:15.266926 (+ 15us) server_negotiation.cc:407] Connection header received
0929 01:02:15.266981 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:15.266984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:15.267030 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:15.267107 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:02:15.267312 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.268073 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.268858 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.269198 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.269299 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.269545 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:02:15.269617 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:02:15.269686 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:15.269786 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 01:02:15.269829 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 01:02:15.276613 2288 raft_consensus.cc:1261] T 12406677897a4430ae5f3c94f8488b4b P 2ac54ced6dad46969c9b046507046048 [term 1 FOLLOWER]: Refusing update from remote peer 39cdc4e943be4c90a96b54aa8654ca0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:15.276765 2426 raft_consensus.cc:1261] T 12406677897a4430ae5f3c94f8488b4b P 17fc9c40b7a24115b9d935449d67e1d7 [term 1 FOLLOWER]: Refusing update from remote peer 39cdc4e943be4c90a96b54aa8654ca0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:15.277169 2645 consensus_queue.cc:1035] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:15.277354 2647 consensus_queue.cc:1035] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:15.282182 2667 mvcc.cc:205] Tried to move back new op lower bound from 7334476739688550400 to 7334476738648801280. Current Snapshot: MvccSnapshot[applied={T|T < 7334476739688550400}]
I20260929 01:02:15.290292 2666 mvcc.cc:205] Tried to move back new op lower bound from 7334476739688550400 to 7334476738648801280. Current Snapshot: MvccSnapshot[applied={T|T < 7334476739688550400}]
I20260929 01:02:15.299660 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.295573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41805 (local address 127.0.98.2:52927)
0929 01:02:15.295962 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:15.295977 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:15.296094 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:15.296379 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:15.296382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:15.296401 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:02:15.296518 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:15.296523 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.297357 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.297363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:15.298338 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:15.298348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.298967 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.298970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:15.299049 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.299090 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:02:15.299392 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:02:15.299397 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 01:02:15.299464 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 01:02:15.299523 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":87,"threads_started":1}
I20260929 01:02:15.299669 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.295619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52927 (local address 127.0.98.3:41805)
0929 01:02:15.295725 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:15.295729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:15.296053 (+ 324us) server_negotiation.cc:407] Connection header received
0929 01:02:15.296227 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:15.296230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:15.296275 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:15.296368 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:02:15.296619 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.297244 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.298495 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.298866 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.298967 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.299202 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:02:15.299277 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:02:15.299346 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:15.299496 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 01:02:15.299549 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 01:02:15.321238 2130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:15.321422 2130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:15.323969 2130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3: Sending DeleteTablet for 3 replicas of tablet 12406677897a4430ae5f3c94f8488b4b
I20260929 01:02:15.324684 2408 tablet_service.cc:1514] Processing DeleteTablet for tablet 12406677897a4430ae5f3c94f8488b4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55772
I20260929 01:02:15.324723 2268 tablet_service.cc:1514] Processing DeleteTablet for tablet 12406677897a4430ae5f3c94f8488b4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46686
I20260929 01:02:15.325163 2685 tablet_replica.cc:330] T 12406677897a4430ae5f3c94f8488b4b P 2ac54ced6dad46969c9b046507046048: stopping tablet replica
I20260929 01:02:15.325436 2685 raft_consensus.cc:2229] T 12406677897a4430ae5f3c94f8488b4b P 2ac54ced6dad46969c9b046507046048 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:15.325413 2548 tablet_service.cc:1514] Processing DeleteTablet for tablet 12406677897a4430ae5f3c94f8488b4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46748
I20260929 01:02:15.325618 2685 raft_consensus.cc:2258] T 12406677897a4430ae5f3c94f8488b4b P 2ac54ced6dad46969c9b046507046048 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:15.326594 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:02:15.327006 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:02:15.327028 2685 ts_tablet_manager.cc:1905] T 12406677897a4430ae5f3c94f8488b4b P 2ac54ced6dad46969c9b046507046048: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:15.328751 2686 tablet_replica.cc:330] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c: stopping tablet replica
I20260929 01:02:15.329087 2686 raft_consensus.cc:2229] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:02:15.329468 2686 raft_consensus.cc:2258] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:15.330770 2686 ts_tablet_manager.cc:1905] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:15.332988 2268 tablet_service.cc:1467] Processing CreateTablet for tablet 47b22d0f4d0047a4976d3f04265ccb6a (DEFAULT_TABLE table=test-table [id=a33943a832ca4785a05abc2927aade54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:15.333228 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47b22d0f4d0047a4976d3f04265ccb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:15.334232 2685 ts_tablet_manager.cc:1918] T 12406677897a4430ae5f3c94f8488b4b P 2ac54ced6dad46969c9b046507046048: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:02:15.334316 2685 log.cc:1196] T 12406677897a4430ae5f3c94f8488b4b P 2ac54ced6dad46969c9b046507046048: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-0/wal/wals/12406677897a4430ae5f3c94f8488b4b
I20260929 01:02:15.334619 2685 ts_tablet_manager.cc:1939] T 12406677897a4430ae5f3c94f8488b4b P 2ac54ced6dad46969c9b046507046048: Deleting consensus metadata
I20260929 01:02:15.335562 2408 tablet_service.cc:1467] Processing CreateTablet for tablet 47b22d0f4d0047a4976d3f04265ccb6a (DEFAULT_TABLE table=test-table [id=a33943a832ca4785a05abc2927aade54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:15.335810 2408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47b22d0f4d0047a4976d3f04265ccb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:15.338066 2637 tablet_bootstrap.cc:492] T 47b22d0f4d0047a4976d3f04265ccb6a P 17fc9c40b7a24115b9d935449d67e1d7: Bootstrap starting.
I20260929 01:02:15.338198 2548 tablet_service.cc:1467] Processing CreateTablet for tablet 47b22d0f4d0047a4976d3f04265ccb6a (DEFAULT_TABLE table=test-table [id=a33943a832ca4785a05abc2927aade54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:15.338416 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47b22d0f4d0047a4976d3f04265ccb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:15.339120 2637 tablet_bootstrap.cc:654] T 47b22d0f4d0047a4976d3f04265ccb6a P 17fc9c40b7a24115b9d935449d67e1d7: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:15.340556 2638 tablet_bootstrap.cc:492] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048: Bootstrap starting.
I20260929 01:02:15.340695 2639 tablet_bootstrap.cc:492] T 47b22d0f4d0047a4976d3f04265ccb6a P 39cdc4e943be4c90a96b54aa8654ca0c: Bootstrap starting.
I20260929 01:02:15.341439 2638 tablet_bootstrap.cc:654] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:15.341570 2639 tablet_bootstrap.cc:654] T 47b22d0f4d0047a4976d3f04265ccb6a P 39cdc4e943be4c90a96b54aa8654ca0c: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:15.342376 2115 catalog_manager.cc:4841] TS 2ac54ced6dad46969c9b046507046048 (127.0.98.1:45525): tablet 12406677897a4430ae5f3c94f8488b4b (table test-table [id=1fa18920de89472f9c16500560b0c048]) successfully deleted
I20260929 01:02:15.343319 2638 tablet_bootstrap.cc:492] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048: No bootstrap required, opened a new log
I20260929 01:02:15.343397 2638 ts_tablet_manager.cc:1397] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 01:02:15.343833 2638 raft_consensus.cc:348] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:15.343962 2638 raft_consensus.cc:374] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:15.344003 2638 raft_consensus.cc:729] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac54ced6dad46969c9b046507046048, State: Initialized, Role: FOLLOWER
I20260929 01:02:15.344156 2638 consensus_queue.cc:260] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:15.344419 2638 ts_tablet_manager.cc:1428] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:15.346010 2686 ts_tablet_manager.cc:1918] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:02:15.346160 2686 log.cc:1196] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/12406677897a4430ae5f3c94f8488b4b
I20260929 01:02:15.346540 2686 ts_tablet_manager.cc:1939] T 12406677897a4430ae5f3c94f8488b4b P 39cdc4e943be4c90a96b54aa8654ca0c: Deleting consensus metadata
I20260929 01:02:15.347697 2639 tablet_bootstrap.cc:492] T 47b22d0f4d0047a4976d3f04265ccb6a P 39cdc4e943be4c90a96b54aa8654ca0c: No bootstrap required, opened a new log
I20260929 01:02:15.347781 2639 ts_tablet_manager.cc:1397] T 47b22d0f4d0047a4976d3f04265ccb6a P 39cdc4e943be4c90a96b54aa8654ca0c: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260929 01:02:15.347990 2684 tablet_replica.cc:330] T 12406677897a4430ae5f3c94f8488b4b P 17fc9c40b7a24115b9d935449d67e1d7: stopping tablet replica
I20260929 01:02:15.348201 2639 raft_consensus.cc:348] T 47b22d0f4d0047a4976d3f04265ccb6a P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:15.348313 2639 raft_consensus.cc:374] T 47b22d0f4d0047a4976d3f04265ccb6a P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:15.348351 2639 raft_consensus.cc:729] T 47b22d0f4d0047a4976d3f04265ccb6a P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39cdc4e943be4c90a96b54aa8654ca0c, State: Initialized, Role: FOLLOWER
I20260929 01:02:15.348479 2639 consensus_queue.cc:260] T 47b22d0f4d0047a4976d3f04265ccb6a P 39cdc4e943be4c90a96b54aa8654ca0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:15.348722 2639 ts_tablet_manager.cc:1428] T 47b22d0f4d0047a4976d3f04265ccb6a P 39cdc4e943be4c90a96b54aa8654ca0c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:15.348976 2684 raft_consensus.cc:2229] T 12406677897a4430ae5f3c94f8488b4b P 17fc9c40b7a24115b9d935449d67e1d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:15.349725 2684 raft_consensus.cc:2258] T 12406677897a4430ae5f3c94f8488b4b P 17fc9c40b7a24115b9d935449d67e1d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:15.349956 2115 catalog_manager.cc:4841] TS 39cdc4e943be4c90a96b54aa8654ca0c (127.0.98.3:41805): tablet 12406677897a4430ae5f3c94f8488b4b (table test-table [id=1fa18920de89472f9c16500560b0c048]) successfully deleted
I20260929 01:02:15.351310 2637 tablet_bootstrap.cc:492] T 47b22d0f4d0047a4976d3f04265ccb6a P 17fc9c40b7a24115b9d935449d67e1d7: No bootstrap required, opened a new log
I20260929 01:02:15.351393 2637 ts_tablet_manager.cc:1397] T 47b22d0f4d0047a4976d3f04265ccb6a P 17fc9c40b7a24115b9d935449d67e1d7: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260929 01:02:15.351778 2684 ts_tablet_manager.cc:1905] T 12406677897a4430ae5f3c94f8488b4b P 17fc9c40b7a24115b9d935449d67e1d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:15.351854 2637 raft_consensus.cc:348] T 47b22d0f4d0047a4976d3f04265ccb6a P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:15.351989 2637 raft_consensus.cc:374] T 47b22d0f4d0047a4976d3f04265ccb6a P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:15.352025 2637 raft_consensus.cc:729] T 47b22d0f4d0047a4976d3f04265ccb6a P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17fc9c40b7a24115b9d935449d67e1d7, State: Initialized, Role: FOLLOWER
I20260929 01:02:15.352146 2637 consensus_queue.cc:260] T 47b22d0f4d0047a4976d3f04265ccb6a P 17fc9c40b7a24115b9d935449d67e1d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:15.352401 2637 ts_tablet_manager.cc:1428] T 47b22d0f4d0047a4976d3f04265ccb6a P 17fc9c40b7a24115b9d935449d67e1d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:15.354741 2684 ts_tablet_manager.cc:1918] T 12406677897a4430ae5f3c94f8488b4b P 17fc9c40b7a24115b9d935449d67e1d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:02:15.354859 2684 log.cc:1196] T 12406677897a4430ae5f3c94f8488b4b P 17fc9c40b7a24115b9d935449d67e1d7: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-1/wal/wals/12406677897a4430ae5f3c94f8488b4b
I20260929 01:02:15.355218 2684 ts_tablet_manager.cc:1939] T 12406677897a4430ae5f3c94f8488b4b P 17fc9c40b7a24115b9d935449d67e1d7: Deleting consensus metadata
I20260929 01:02:15.356189 2115 catalog_manager.cc:4841] TS 17fc9c40b7a24115b9d935449d67e1d7 (127.0.98.2:34141): tablet 12406677897a4430ae5f3c94f8488b4b (table test-table [id=1fa18920de89472f9c16500560b0c048]) successfully deleted
I20260929 01:02:15.396167 2644 raft_consensus.cc:482] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:02:15.396306 2644 raft_consensus.cc:504] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:15.397589 2644 leader_election.cc:288] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17fc9c40b7a24115b9d935449d67e1d7 (127.0.98.2:34141), 39cdc4e943be4c90a96b54aa8654ca0c (127.0.98.3:41805)
I20260929 01:02:15.402714 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.397917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56593 (local address 127.0.98.2:34141)
0929 01:02:15.398131 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:15.398137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:15.398462 (+ 325us) server_negotiation.cc:407] Connection header received
0929 01:02:15.398701 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:15.398704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:15.398762 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:15.398857 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:15.399353 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.400070 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.401622 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.402078 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.402227 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.402305 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:15.402521 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 01:02:15.402593 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 01:02:15.402748 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.397746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34141 (local address 127.0.98.1:56593)
0929 01:02:15.398352 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:15.398371 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:15.398548 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:15.399024 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:15.399027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:15.399047 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:15.399193 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:15.399201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.400215 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.400219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:15.401492 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:15.401503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.402244 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.402248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:15.402361 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.402392 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 01:02:15.402488 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 01:02:15.402568 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":65,"threads_started":1}
I20260929 01:02:15.403493 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b22d0f4d0047a4976d3f04265ccb6a" candidate_uuid: "2ac54ced6dad46969c9b046507046048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17fc9c40b7a24115b9d935449d67e1d7" is_pre_election: true
I20260929 01:02:15.403668 2426 raft_consensus.cc:2454] T 47b22d0f4d0047a4976d3f04265ccb6a P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ac54ced6dad46969c9b046507046048 in term 0.
I20260929 01:02:15.403878 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.398565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38577 (local address 127.0.98.3:41805)
0929 01:02:15.398697 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:15.398702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:15.399502 (+ 800us) server_negotiation.cc:407] Connection header received
0929 01:02:15.399694 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:15.399698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:15.399748 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:15.399818 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:15.400302 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.401075 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.402583 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.403206 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.403283 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.403341 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:15.403691 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 01:02:15.403766 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 01:02:15.404073 2219 leader_election.cc:302] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17fc9c40b7a24115b9d935449d67e1d7, 2ac54ced6dad46969c9b046507046048; no voters:
I20260929 01:02:15.404230 2691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.398191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41805 (local address 127.0.98.1:38577)
0929 01:02:15.399416 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:15.399430 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:15.399569 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:15.399949 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:15.399952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:15.399968 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:15.400101 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:15.400111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.401203 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.401206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:15.402421 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:15.402432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.403442 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.403445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:15.403555 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.403576 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 01:02:15.404007 (+ 431us) client_negotiation.cc:240] Negotiation successful
0929 01:02:15.404097 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"thread_start_us":97,"threads_started":1}
I20260929 01:02:15.404626 2644 raft_consensus.cc:2781] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:02:15.404700 2644 raft_consensus.cc:482] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:02:15.404703 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b22d0f4d0047a4976d3f04265ccb6a" candidate_uuid: "2ac54ced6dad46969c9b046507046048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" is_pre_election: true
I20260929 01:02:15.404747 2644 raft_consensus.cc:3037] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:15.404821 2568 raft_consensus.cc:2454] T 47b22d0f4d0047a4976d3f04265ccb6a P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ac54ced6dad46969c9b046507046048 in term 0.
I20260929 01:02:15.405838 2644 raft_consensus.cc:504] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:15.406240 2644 leader_election.cc:288] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048 [CANDIDATE]: Term 1 election: Requested vote from peers 17fc9c40b7a24115b9d935449d67e1d7 (127.0.98.2:34141), 39cdc4e943be4c90a96b54aa8654ca0c (127.0.98.3:41805)
I20260929 01:02:15.406529 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b22d0f4d0047a4976d3f04265ccb6a" candidate_uuid: "2ac54ced6dad46969c9b046507046048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17fc9c40b7a24115b9d935449d67e1d7"
I20260929 01:02:15.406634 2426 raft_consensus.cc:3037] T 47b22d0f4d0047a4976d3f04265ccb6a P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:15.406600 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b22d0f4d0047a4976d3f04265ccb6a" candidate_uuid: "2ac54ced6dad46969c9b046507046048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39cdc4e943be4c90a96b54aa8654ca0c"
I20260929 01:02:15.406723 2568 raft_consensus.cc:3037] T 47b22d0f4d0047a4976d3f04265ccb6a P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:15.407676 2568 raft_consensus.cc:2454] T 47b22d0f4d0047a4976d3f04265ccb6a P 39cdc4e943be4c90a96b54aa8654ca0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ac54ced6dad46969c9b046507046048 in term 1.
I20260929 01:02:15.407712 2426 raft_consensus.cc:2454] T 47b22d0f4d0047a4976d3f04265ccb6a P 17fc9c40b7a24115b9d935449d67e1d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ac54ced6dad46969c9b046507046048 in term 1.
I20260929 01:02:15.407968 2219 leader_election.cc:302] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17fc9c40b7a24115b9d935449d67e1d7, 2ac54ced6dad46969c9b046507046048; no voters:
I20260929 01:02:15.408149 2644 raft_consensus.cc:2781] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:15.408420 2644 raft_consensus.cc:686] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048 [term 1 LEADER]: Becoming Leader. State: Replica: 2ac54ced6dad46969c9b046507046048, State: Running, Role: LEADER
I20260929 01:02:15.408674 2644 consensus_queue.cc:237] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:15.411468 2130 catalog_manager.cc:5495] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ac54ced6dad46969c9b046507046048 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2ac54ced6dad46969c9b046507046048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } health_report { overall_health: UNKNOWN } } }
I20260929 01:02:15.445955 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.442078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46698 (local address 127.0.98.1:45525)
0929 01:02:15.442410 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:15.442413 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:15.442423 (+ 10us) server_negotiation.cc:407] Connection header received
0929 01:02:15.442480 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:15.442482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:15.442525 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:15.442636 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:02:15.442895 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.443414 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.444318 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.444615 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.444724 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.445037 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:02:15.445484 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:02:15.445614 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:15.445716 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 01:02:15.445771 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270}
I20260929 01:02:15.456377 2696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.452891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34141 (local address 127.0.98.1:50529)
0929 01:02:15.453174 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:15.453188 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:15.453278 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:15.453542 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:15.453545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:15.453553 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:02:15.453661 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:15.453671 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.454403 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.454408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:15.455093 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:15.455102 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.455671 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.455675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:15.455762 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.455809 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:02:15.456113 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:02:15.456121 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 01:02:15.456182 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 01:02:15.456257 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":107,"threads_started":1}
I20260929 01:02:15.456372 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.453190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50529 (local address 127.0.98.2:34141)
0929 01:02:15.453310 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:15.453316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:15.453342 (+ 26us) server_negotiation.cc:407] Connection header received
0929 01:02:15.453393 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:15.453395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:15.453446 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:15.453525 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:02:15.453794 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.454290 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.455217 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.455559 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.455673 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.455945 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:02:15.456019 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:02:15.456099 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:15.456191 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 01:02:15.456236 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:02:15.462308 2568 raft_consensus.cc:1261] T 47b22d0f4d0047a4976d3f04265ccb6a P 39cdc4e943be4c90a96b54aa8654ca0c [term 1 FOLLOWER]: Refusing update from remote peer 2ac54ced6dad46969c9b046507046048: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:15.462389 2426 raft_consensus.cc:1261] T 47b22d0f4d0047a4976d3f04265ccb6a P 17fc9c40b7a24115b9d935449d67e1d7 [term 1 FOLLOWER]: Refusing update from remote peer 2ac54ced6dad46969c9b046507046048: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:15.462824 2692 consensus_queue.cc:1035] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:15.463011 2644 consensus_queue.cc:1035] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:15.481065 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.477551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41895 (local address 127.0.98.1:45525)
0929 01:02:15.477652 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:15.477656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:15.477715 (+ 59us) server_negotiation.cc:407] Connection header received
0929 01:02:15.477852 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:15.477854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:15.477897 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:15.477972 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:02:15.478248 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.478723 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.479884 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.480205 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.480324 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.480648 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:02:15.480727 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:02:15.480786 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:15.480917 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 01:02:15.480957 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 01:02:15.481132 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:15.477499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45525 (local address 127.0.98.2:41895)
0929 01:02:15.477659 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:15.477674 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:15.477757 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:15.478005 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:15.478008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:15.478017 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:02:15.478119 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:15.478125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.478852 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.478856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:15.479743 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:15.479753 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:15.480404 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:15.480408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:15.480505 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:15.480536 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:02:15.480822 (+ 286us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:02:15.480830 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 01:02:15.480913 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 01:02:15.480997 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
W20260929 01:02:15.481721 2666 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 01:02:17.037922 2131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:17.038080 2131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:17.040417 2131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3: Sending DeleteTablet for 3 replicas of tablet 47b22d0f4d0047a4976d3f04265ccb6a
I20260929 01:02:17.040962 2268 tablet_service.cc:1514] Processing DeleteTablet for tablet 47b22d0f4d0047a4976d3f04265ccb6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46686
I20260929 01:02:17.041074 2408 tablet_service.cc:1514] Processing DeleteTablet for tablet 47b22d0f4d0047a4976d3f04265ccb6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55772
I20260929 01:02:17.041286 2548 tablet_service.cc:1514] Processing DeleteTablet for tablet 47b22d0f4d0047a4976d3f04265ccb6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46748
I20260929 01:02:17.041330 2725 tablet_replica.cc:330] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048: stopping tablet replica
I20260929 01:02:17.041463 2725 raft_consensus.cc:2229] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:02:17.041641 2727 tablet_replica.cc:330] T 47b22d0f4d0047a4976d3f04265ccb6a P 39cdc4e943be4c90a96b54aa8654ca0c: stopping tablet replica
I20260929 01:02:17.041795 2726 tablet_replica.cc:330] T 47b22d0f4d0047a4976d3f04265ccb6a P 17fc9c40b7a24115b9d935449d67e1d7: stopping tablet replica
I20260929 01:02:17.041786 2727 raft_consensus.cc:2229] T 47b22d0f4d0047a4976d3f04265ccb6a P 39cdc4e943be4c90a96b54aa8654ca0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:17.041782 2725 raft_consensus.cc:2258] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:17.041905 2727 raft_consensus.cc:2258] T 47b22d0f4d0047a4976d3f04265ccb6a P 39cdc4e943be4c90a96b54aa8654ca0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:17.042178 2726 raft_consensus.cc:2229] T 47b22d0f4d0047a4976d3f04265ccb6a P 17fc9c40b7a24115b9d935449d67e1d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:17.042299 2726 raft_consensus.cc:2258] T 47b22d0f4d0047a4976d3f04265ccb6a P 17fc9c40b7a24115b9d935449d67e1d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:17.042361 2725 ts_tablet_manager.cc:1905] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:17.042878 2726 ts_tablet_manager.cc:1905] T 47b22d0f4d0047a4976d3f04265ccb6a P 17fc9c40b7a24115b9d935449d67e1d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:17.045189 2727 ts_tablet_manager.cc:1905] T 47b22d0f4d0047a4976d3f04265ccb6a P 39cdc4e943be4c90a96b54aa8654ca0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:17.045418 2725 ts_tablet_manager.cc:1918] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 01:02:17.045503 2725 log.cc:1196] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-0/wal/wals/47b22d0f4d0047a4976d3f04265ccb6a
I20260929 01:02:17.045821 2725 ts_tablet_manager.cc:1939] T 47b22d0f4d0047a4976d3f04265ccb6a P 2ac54ced6dad46969c9b046507046048: Deleting consensus metadata
I20260929 01:02:17.046495 2115 catalog_manager.cc:4841] TS 2ac54ced6dad46969c9b046507046048 (127.0.98.1:45525): tablet 47b22d0f4d0047a4976d3f04265ccb6a (table test-table [id=a33943a832ca4785a05abc2927aade54]) successfully deleted
I20260929 01:02:17.046780 2726 ts_tablet_manager.cc:1918] T 47b22d0f4d0047a4976d3f04265ccb6a P 17fc9c40b7a24115b9d935449d67e1d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 01:02:17.046893 2726 log.cc:1196] T 47b22d0f4d0047a4976d3f04265ccb6a P 17fc9c40b7a24115b9d935449d67e1d7: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-1/wal/wals/47b22d0f4d0047a4976d3f04265ccb6a
I20260929 01:02:17.047176 2726 ts_tablet_manager.cc:1939] T 47b22d0f4d0047a4976d3f04265ccb6a P 17fc9c40b7a24115b9d935449d67e1d7: Deleting consensus metadata
I20260929 01:02:17.047586 2727 ts_tablet_manager.cc:1918] T 47b22d0f4d0047a4976d3f04265ccb6a P 39cdc4e943be4c90a96b54aa8654ca0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 01:02:17.047673 2727 log.cc:1196] T 47b22d0f4d0047a4976d3f04265ccb6a P 39cdc4e943be4c90a96b54aa8654ca0c: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/47b22d0f4d0047a4976d3f04265ccb6a
I20260929 01:02:17.047782 2115 catalog_manager.cc:4841] TS 17fc9c40b7a24115b9d935449d67e1d7 (127.0.98.2:34141): tablet 47b22d0f4d0047a4976d3f04265ccb6a (table test-table [id=a33943a832ca4785a05abc2927aade54]) successfully deleted
I20260929 01:02:17.047956 2727 ts_tablet_manager.cc:1939] T 47b22d0f4d0047a4976d3f04265ccb6a P 39cdc4e943be4c90a96b54aa8654ca0c: Deleting consensus metadata
I20260929 01:02:17.048635 2115 catalog_manager.cc:4841] TS 39cdc4e943be4c90a96b54aa8654ca0c (127.0.98.3:41805): tablet 47b22d0f4d0047a4976d3f04265ccb6a (table test-table [id=a33943a832ca4785a05abc2927aade54]) successfully deleted
Sep 29 01:02:17 dist-test-slave-7zdl krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643737, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 01:02:17 dist-test-slave-7zdl krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643737, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:17.074884 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:17.061651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47726 (local address 127.0.98.62:39297)
0929 01:02:17.061909 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:17.061921 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:17.061983 (+ 62us) server_negotiation.cc:407] Connection header received
0929 01:02:17.062140 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:17.062143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:17.062197 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:17.062291 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:17.063036 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:17.063594 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:17.064429 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:17.064668 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:17.064777 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:17.071597 (+ 6820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:17.071601 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:17.071603 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:17.071627 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:17.073147 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:17.073570 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:17.073574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:17.073576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:17.073659 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:17.073930 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:17.073935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:17.073937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:17.074127 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:17.074267 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:17.074510 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 01:02:17.074634 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":84,"threads_started":1}
I20260929 01:02:17.077402 2131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:02:17.077842 2131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:02:17.082290 2268 tablet_service.cc:1467] Processing CreateTablet for tablet 220a063f354543b68224165b050c8911 (DEFAULT_TABLE table=test-table [id=c440adc4a229457594732a4623981ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:17.082319 2548 tablet_service.cc:1467] Processing CreateTablet for tablet 220a063f354543b68224165b050c8911 (DEFAULT_TABLE table=test-table [id=c440adc4a229457594732a4623981ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:17.082331 2408 tablet_service.cc:1467] Processing CreateTablet for tablet 220a063f354543b68224165b050c8911 (DEFAULT_TABLE table=test-table [id=c440adc4a229457594732a4623981ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:17.082589 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220a063f354543b68224165b050c8911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:17.082587 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220a063f354543b68224165b050c8911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:17.082643 2408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220a063f354543b68224165b050c8911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:17.085148 2738 tablet_bootstrap.cc:492] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7: Bootstrap starting.
I20260929 01:02:17.085613 2736 tablet_bootstrap.cc:492] T 220a063f354543b68224165b050c8911 P 39cdc4e943be4c90a96b54aa8654ca0c: Bootstrap starting.
I20260929 01:02:17.086056 2738 tablet_bootstrap.cc:654] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:17.086513 2736 tablet_bootstrap.cc:654] T 220a063f354543b68224165b050c8911 P 39cdc4e943be4c90a96b54aa8654ca0c: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:17.087193 2737 tablet_bootstrap.cc:492] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048: Bootstrap starting.
I20260929 01:02:17.088003 2737 tablet_bootstrap.cc:654] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:17.088271 2738 tablet_bootstrap.cc:492] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7: No bootstrap required, opened a new log
I20260929 01:02:17.088368 2738 ts_tablet_manager.cc:1397] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 01:02:17.088887 2738 raft_consensus.cc:348] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:17.089010 2738 raft_consensus.cc:374] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:17.089054 2736 tablet_bootstrap.cc:492] T 220a063f354543b68224165b050c8911 P 39cdc4e943be4c90a96b54aa8654ca0c: No bootstrap required, opened a new log
I20260929 01:02:17.089054 2738 raft_consensus.cc:729] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17fc9c40b7a24115b9d935449d67e1d7, State: Initialized, Role: FOLLOWER
I20260929 01:02:17.089131 2736 ts_tablet_manager.cc:1397] T 220a063f354543b68224165b050c8911 P 39cdc4e943be4c90a96b54aa8654ca0c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 01:02:17.089170 2737 tablet_bootstrap.cc:492] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048: No bootstrap required, opened a new log
I20260929 01:02:17.089244 2737 ts_tablet_manager.cc:1397] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 01:02:17.089208 2738 consensus_queue.cc:260] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:17.089506 2738 ts_tablet_manager.cc:1428] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:17.089529 2736 raft_consensus.cc:348] T 220a063f354543b68224165b050c8911 P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:17.089630 2736 raft_consensus.cc:374] T 220a063f354543b68224165b050c8911 P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:17.089661 2736 raft_consensus.cc:729] T 220a063f354543b68224165b050c8911 P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39cdc4e943be4c90a96b54aa8654ca0c, State: Initialized, Role: FOLLOWER
I20260929 01:02:17.089635 2737 raft_consensus.cc:348] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:17.089727 2737 raft_consensus.cc:374] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:17.089783 2737 raft_consensus.cc:729] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac54ced6dad46969c9b046507046048, State: Initialized, Role: FOLLOWER
I20260929 01:02:17.089780 2736 consensus_queue.cc:260] T 220a063f354543b68224165b050c8911 P 39cdc4e943be4c90a96b54aa8654ca0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:17.089898 2737 consensus_queue.cc:260] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:17.090147 2736 ts_tablet_manager.cc:1428] T 220a063f354543b68224165b050c8911 P 39cdc4e943be4c90a96b54aa8654ca0c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:17.090553 2737 ts_tablet_manager.cc:1428] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:17.188910 2702 raft_consensus.cc:482] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:02:17.189060 2702 raft_consensus.cc:504] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:17.189596 2702 leader_election.cc:288] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ac54ced6dad46969c9b046507046048 (127.0.98.1:45525), 39cdc4e943be4c90a96b54aa8654ca0c (127.0.98.3:41805)
I20260929 01:02:17.189857 2716 raft_consensus.cc:482] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:02:17.189986 2716 raft_consensus.cc:504] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:17.190066 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220a063f354543b68224165b050c8911" candidate_uuid: "17fc9c40b7a24115b9d935449d67e1d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac54ced6dad46969c9b046507046048" is_pre_election: true
I20260929 01:02:17.190202 2287 raft_consensus.cc:2454] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17fc9c40b7a24115b9d935449d67e1d7 in term 0.
I20260929 01:02:17.190387 2716 leader_election.cc:288] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17fc9c40b7a24115b9d935449d67e1d7 (127.0.98.2:34141), 39cdc4e943be4c90a96b54aa8654ca0c (127.0.98.3:41805)
I20260929 01:02:17.190656 2359 leader_election.cc:302] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17fc9c40b7a24115b9d935449d67e1d7, 2ac54ced6dad46969c9b046507046048; no voters:
I20260929 01:02:17.190718 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220a063f354543b68224165b050c8911" candidate_uuid: "2ac54ced6dad46969c9b046507046048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17fc9c40b7a24115b9d935449d67e1d7" is_pre_election: true
I20260929 01:02:17.190856 2426 raft_consensus.cc:2454] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ac54ced6dad46969c9b046507046048 in term 0.
I20260929 01:02:17.190830 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220a063f354543b68224165b050c8911" candidate_uuid: "2ac54ced6dad46969c9b046507046048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" is_pre_election: true
I20260929 01:02:17.190924 2702 raft_consensus.cc:2781] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:02:17.190826 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220a063f354543b68224165b050c8911" candidate_uuid: "17fc9c40b7a24115b9d935449d67e1d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" is_pre_election: true
I20260929 01:02:17.190973 2702 raft_consensus.cc:482] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:02:17.191009 2702 raft_consensus.cc:3037] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:17.191001 2567 raft_consensus.cc:2454] T 220a063f354543b68224165b050c8911 P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ac54ced6dad46969c9b046507046048 in term 0.
I20260929 01:02:17.191155 2219 leader_election.cc:302] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17fc9c40b7a24115b9d935449d67e1d7, 2ac54ced6dad46969c9b046507046048; no voters:
I20260929 01:02:17.191318 2716 raft_consensus.cc:2781] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:02:17.191372 2716 raft_consensus.cc:482] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:02:17.191401 2716 raft_consensus.cc:3037] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:17.192175 2702 raft_consensus.cc:504] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:17.192376 2716 raft_consensus.cc:504] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:17.192512 2702 leader_election.cc:288] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7 [CANDIDATE]: Term 1 election: Requested vote from peers 2ac54ced6dad46969c9b046507046048 (127.0.98.1:45525), 39cdc4e943be4c90a96b54aa8654ca0c (127.0.98.3:41805)
I20260929 01:02:17.192703 2716 leader_election.cc:288] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048 [CANDIDATE]: Term 1 election: Requested vote from peers 17fc9c40b7a24115b9d935449d67e1d7 (127.0.98.2:34141), 39cdc4e943be4c90a96b54aa8654ca0c (127.0.98.3:41805)
I20260929 01:02:17.193051 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220a063f354543b68224165b050c8911" candidate_uuid: "17fc9c40b7a24115b9d935449d67e1d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39cdc4e943be4c90a96b54aa8654ca0c"
I20260929 01:02:17.193071 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220a063f354543b68224165b050c8911" candidate_uuid: "2ac54ced6dad46969c9b046507046048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17fc9c40b7a24115b9d935449d67e1d7"
I20260929 01:02:17.193123 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220a063f354543b68224165b050c8911" candidate_uuid: "17fc9c40b7a24115b9d935449d67e1d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ac54ced6dad46969c9b046507046048"
I20260929 01:02:17.193166 2567 raft_consensus.cc:3037] T 220a063f354543b68224165b050c8911 P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:17.193223 2426 raft_consensus.cc:2379] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2ac54ced6dad46969c9b046507046048 in current term 1: Already voted for candidate 17fc9c40b7a24115b9d935449d67e1d7 in this term.
I20260929 01:02:17.193292 2287 raft_consensus.cc:2379] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 17fc9c40b7a24115b9d935449d67e1d7 in current term 1: Already voted for candidate 2ac54ced6dad46969c9b046507046048 in this term.
I20260929 01:02:17.193501 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220a063f354543b68224165b050c8911" candidate_uuid: "2ac54ced6dad46969c9b046507046048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39cdc4e943be4c90a96b54aa8654ca0c"
I20260929 01:02:17.193819 2219 leader_election.cc:302] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ac54ced6dad46969c9b046507046048; no voters: 17fc9c40b7a24115b9d935449d67e1d7, 39cdc4e943be4c90a96b54aa8654ca0c
I20260929 01:02:17.194008 2716 raft_consensus.cc:2726] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 01:02:17.194381 2567 raft_consensus.cc:2454] T 220a063f354543b68224165b050c8911 P 39cdc4e943be4c90a96b54aa8654ca0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17fc9c40b7a24115b9d935449d67e1d7 in term 1.
I20260929 01:02:17.194618 2359 leader_election.cc:302] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17fc9c40b7a24115b9d935449d67e1d7, 39cdc4e943be4c90a96b54aa8654ca0c; no voters: 2ac54ced6dad46969c9b046507046048
I20260929 01:02:17.194813 2702 raft_consensus.cc:2781] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:17.194929 2702 raft_consensus.cc:686] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7 [term 1 LEADER]: Becoming Leader. State: Replica: 17fc9c40b7a24115b9d935449d67e1d7, State: Running, Role: LEADER
I20260929 01:02:17.195106 2702 consensus_queue.cc:237] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:17.196695 2129 catalog_manager.cc:5495] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17fc9c40b7a24115b9d935449d67e1d7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "17fc9c40b7a24115b9d935449d67e1d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } health_report { overall_health: UNKNOWN } } }
I20260929 01:02:17.213786 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:17.210525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55792 (local address 127.0.98.2:34141)
0929 01:02:17.210787 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:17.210811 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:17.210825 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:02:17.210872 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:17.210874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:17.210912 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:17.211003 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:02:17.211251 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:17.211781 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:17.212746 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:17.212988 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:17.213096 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:17.213342 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:02:17.213418 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:02:17.213498 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:17.213590 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 01:02:17.213637 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":100,"threads_started":1}
I20260929 01:02:17.222265 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:17.219281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33817 (local address 127.0.98.2:34141)
0929 01:02:17.219389 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:17.219393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:17.219478 (+ 85us) server_negotiation.cc:407] Connection header received
0929 01:02:17.219622 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:17.219625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:17.219669 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:17.219745 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:02:17.219984 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:17.220477 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:17.221294 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:17.221504 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:17.221610 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:17.221857 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:02:17.221922 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:02:17.222000 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:17.222086 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 01:02:17.222124 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":41}
I20260929 01:02:17.222266 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:17.219136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34141 (local address 127.0.98.2:33817)
0929 01:02:17.219391 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:17.219405 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:17.219527 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:17.219752 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:17.219755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:17.219761 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:02:17.219877 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:17.219882 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:17.220594 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:17.220600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:17.221184 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:17.221191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:17.221608 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:17.221611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:17.221711 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:17.221740 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:02:17.222002 (+ 262us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:02:17.222009 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 01:02:17.222069 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 01:02:17.222147 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":25,"thread_start_us":95,"threads_started":1}
I20260929 01:02:17.227624 2567 raft_consensus.cc:1261] T 220a063f354543b68224165b050c8911 P 39cdc4e943be4c90a96b54aa8654ca0c [term 1 FOLLOWER]: Refusing update from remote peer 17fc9c40b7a24115b9d935449d67e1d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:17.227665 2287 raft_consensus.cc:1261] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048 [term 1 FOLLOWER]: Refusing update from remote peer 17fc9c40b7a24115b9d935449d67e1d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:17.227993 2706 consensus_queue.cc:1035] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:17.228142 2702 consensus_queue.cc:1035] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:17.259591 2130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:17.259778 2130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:17.261806 2130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3: Sending DeleteTablet for 3 replicas of tablet 220a063f354543b68224165b050c8911
I20260929 01:02:17.262549 2268 tablet_service.cc:1514] Processing DeleteTablet for tablet 220a063f354543b68224165b050c8911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46686
I20260929 01:02:17.262742 2548 tablet_service.cc:1514] Processing DeleteTablet for tablet 220a063f354543b68224165b050c8911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46748
I20260929 01:02:17.262912 2725 tablet_replica.cc:330] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048: stopping tablet replica
I20260929 01:02:17.262939 2727 tablet_replica.cc:330] T 220a063f354543b68224165b050c8911 P 39cdc4e943be4c90a96b54aa8654ca0c: stopping tablet replica
I20260929 01:02:17.263054 2727 raft_consensus.cc:2229] T 220a063f354543b68224165b050c8911 P 39cdc4e943be4c90a96b54aa8654ca0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:17.263054 2725 raft_consensus.cc:2229] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:17.263021 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 01:02:17.263156 2727 raft_consensus.cc:2258] T 220a063f354543b68224165b050c8911 P 39cdc4e943be4c90a96b54aa8654ca0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:17.263159 2725 raft_consensus.cc:2258] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 01:02:17.263417 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:02:17.263705 2407 tablet_service.cc:1514] Processing DeleteTablet for tablet 220a063f354543b68224165b050c8911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55772
I20260929 01:02:17.263875 2726 tablet_replica.cc:330] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7: stopping tablet replica
I20260929 01:02:17.263880 2725 ts_tablet_manager.cc:1905] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:17.263880 2727 ts_tablet_manager.cc:1905] T 220a063f354543b68224165b050c8911 P 39cdc4e943be4c90a96b54aa8654ca0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:17.263970 2726 raft_consensus.cc:2229] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:02:17.264241 2726 raft_consensus.cc:2258] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:17.265569 2726 ts_tablet_manager.cc:1905] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:17.266363 2727 ts_tablet_manager.cc:1918] T 220a063f354543b68224165b050c8911 P 39cdc4e943be4c90a96b54aa8654ca0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:02:17.266362 2725 ts_tablet_manager.cc:1918] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:02:17.266443 2725 log.cc:1196] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-0/wal/wals/220a063f354543b68224165b050c8911
I20260929 01:02:17.266443 2727 log.cc:1196] T 220a063f354543b68224165b050c8911 P 39cdc4e943be4c90a96b54aa8654ca0c: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/220a063f354543b68224165b050c8911
I20260929 01:02:17.266714 2725 ts_tablet_manager.cc:1939] T 220a063f354543b68224165b050c8911 P 2ac54ced6dad46969c9b046507046048: Deleting consensus metadata
I20260929 01:02:17.266716 2727 ts_tablet_manager.cc:1939] T 220a063f354543b68224165b050c8911 P 39cdc4e943be4c90a96b54aa8654ca0c: Deleting consensus metadata
I20260929 01:02:17.267388 2115 catalog_manager.cc:4841] TS 39cdc4e943be4c90a96b54aa8654ca0c (127.0.98.3:41805): tablet 220a063f354543b68224165b050c8911 (table test-table [id=c440adc4a229457594732a4623981ffa]) successfully deleted
I20260929 01:02:17.267556 2115 catalog_manager.cc:4841] TS 2ac54ced6dad46969c9b046507046048 (127.0.98.1:45525): tablet 220a063f354543b68224165b050c8911 (table test-table [id=c440adc4a229457594732a4623981ffa]) successfully deleted
I20260929 01:02:17.267990 2726 ts_tablet_manager.cc:1918] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:02:17.268082 2726 log.cc:1196] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-1/wal/wals/220a063f354543b68224165b050c8911
I20260929 01:02:17.268347 2726 ts_tablet_manager.cc:1939] T 220a063f354543b68224165b050c8911 P 17fc9c40b7a24115b9d935449d67e1d7: Deleting consensus metadata
I20260929 01:02:17.268953 2115 catalog_manager.cc:4841] TS 17fc9c40b7a24115b9d935449d67e1d7 (127.0.98.2:34141): tablet 220a063f354543b68224165b050c8911 (table test-table [id=c440adc4a229457594732a4623981ffa]) successfully deleted
I20260929 01:02:17.271646 2268 tablet_service.cc:1467] Processing CreateTablet for tablet db843963cd6e4e8bb352a53772624359 (DEFAULT_TABLE table=test-table [id=bfb3bdddd53a48a8951c1c05ba0a2b85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:17.271760 2407 tablet_service.cc:1467] Processing CreateTablet for tablet db843963cd6e4e8bb352a53772624359 (DEFAULT_TABLE table=test-table [id=bfb3bdddd53a48a8951c1c05ba0a2b85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:17.271898 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db843963cd6e4e8bb352a53772624359. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:17.272014 2407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db843963cd6e4e8bb352a53772624359. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:17.273707 2738 tablet_bootstrap.cc:492] T db843963cd6e4e8bb352a53772624359 P 17fc9c40b7a24115b9d935449d67e1d7: Bootstrap starting.
I20260929 01:02:17.274003 2737 tablet_bootstrap.cc:492] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048: Bootstrap starting.
I20260929 01:02:17.274459 2738 tablet_bootstrap.cc:654] T db843963cd6e4e8bb352a53772624359 P 17fc9c40b7a24115b9d935449d67e1d7: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:17.274760 2737 tablet_bootstrap.cc:654] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:17.275043 2548 tablet_service.cc:1467] Processing CreateTablet for tablet db843963cd6e4e8bb352a53772624359 (DEFAULT_TABLE table=test-table [id=bfb3bdddd53a48a8951c1c05ba0a2b85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:17.275274 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db843963cd6e4e8bb352a53772624359. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:17.275496 2738 tablet_bootstrap.cc:492] T db843963cd6e4e8bb352a53772624359 P 17fc9c40b7a24115b9d935449d67e1d7: No bootstrap required, opened a new log
I20260929 01:02:17.275595 2738 ts_tablet_manager.cc:1397] T db843963cd6e4e8bb352a53772624359 P 17fc9c40b7a24115b9d935449d67e1d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 01:02:17.275920 2738 raft_consensus.cc:348] T db843963cd6e4e8bb352a53772624359 P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:17.276041 2738 raft_consensus.cc:374] T db843963cd6e4e8bb352a53772624359 P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:17.276081 2738 raft_consensus.cc:729] T db843963cd6e4e8bb352a53772624359 P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17fc9c40b7a24115b9d935449d67e1d7, State: Initialized, Role: FOLLOWER
I20260929 01:02:17.276203 2738 consensus_queue.cc:260] T db843963cd6e4e8bb352a53772624359 P 17fc9c40b7a24115b9d935449d67e1d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:17.276464 2738 ts_tablet_manager.cc:1428] T db843963cd6e4e8bb352a53772624359 P 17fc9c40b7a24115b9d935449d67e1d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:17.277297 2736 tablet_bootstrap.cc:492] T db843963cd6e4e8bb352a53772624359 P 39cdc4e943be4c90a96b54aa8654ca0c: Bootstrap starting.
I20260929 01:02:17.278147 2736 tablet_bootstrap.cc:654] T db843963cd6e4e8bb352a53772624359 P 39cdc4e943be4c90a96b54aa8654ca0c: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:17.278682 2737 tablet_bootstrap.cc:492] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048: No bootstrap required, opened a new log
I20260929 01:02:17.278759 2737 ts_tablet_manager.cc:1397] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 01:02:17.279151 2737 raft_consensus.cc:348] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:17.279246 2737 raft_consensus.cc:374] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:17.279256 2736 tablet_bootstrap.cc:492] T db843963cd6e4e8bb352a53772624359 P 39cdc4e943be4c90a96b54aa8654ca0c: No bootstrap required, opened a new log
I20260929 01:02:17.279275 2737 raft_consensus.cc:729] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac54ced6dad46969c9b046507046048, State: Initialized, Role: FOLLOWER
I20260929 01:02:17.279314 2736 ts_tablet_manager.cc:1397] T db843963cd6e4e8bb352a53772624359 P 39cdc4e943be4c90a96b54aa8654ca0c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 01:02:17.279381 2737 consensus_queue.cc:260] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:17.279575 2736 raft_consensus.cc:348] T db843963cd6e4e8bb352a53772624359 P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:17.279613 2737 ts_tablet_manager.cc:1428] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:17.279646 2736 raft_consensus.cc:374] T db843963cd6e4e8bb352a53772624359 P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:17.279667 2736 raft_consensus.cc:729] T db843963cd6e4e8bb352a53772624359 P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39cdc4e943be4c90a96b54aa8654ca0c, State: Initialized, Role: FOLLOWER
I20260929 01:02:17.279750 2736 consensus_queue.cc:260] T db843963cd6e4e8bb352a53772624359 P 39cdc4e943be4c90a96b54aa8654ca0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:17.279974 2736 ts_tablet_manager.cc:1428] T db843963cd6e4e8bb352a53772624359 P 39cdc4e943be4c90a96b54aa8654ca0c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 01:02:17.405248 2716 raft_consensus.cc:482] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:02:17.405380 2716 raft_consensus.cc:504] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:17.405725 2716 leader_election.cc:288] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17fc9c40b7a24115b9d935449d67e1d7 (127.0.98.2:34141), 39cdc4e943be4c90a96b54aa8654ca0c (127.0.98.3:41805)
I20260929 01:02:17.406193 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db843963cd6e4e8bb352a53772624359" candidate_uuid: "2ac54ced6dad46969c9b046507046048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17fc9c40b7a24115b9d935449d67e1d7" is_pre_election: true
I20260929 01:02:17.406332 2426 raft_consensus.cc:2454] T db843963cd6e4e8bb352a53772624359 P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ac54ced6dad46969c9b046507046048 in term 0.
I20260929 01:02:17.406282 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db843963cd6e4e8bb352a53772624359" candidate_uuid: "2ac54ced6dad46969c9b046507046048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" is_pre_election: true
I20260929 01:02:17.406500 2567 raft_consensus.cc:2454] T db843963cd6e4e8bb352a53772624359 P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ac54ced6dad46969c9b046507046048 in term 0.
I20260929 01:02:17.406725 2219 leader_election.cc:302] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17fc9c40b7a24115b9d935449d67e1d7, 2ac54ced6dad46969c9b046507046048; no voters:
I20260929 01:02:17.406908 2716 raft_consensus.cc:2781] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:02:17.406973 2716 raft_consensus.cc:482] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:02:17.407022 2716 raft_consensus.cc:3037] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:17.407976 2716 raft_consensus.cc:504] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:17.408293 2716 leader_election.cc:288] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048 [CANDIDATE]: Term 1 election: Requested vote from peers 17fc9c40b7a24115b9d935449d67e1d7 (127.0.98.2:34141), 39cdc4e943be4c90a96b54aa8654ca0c (127.0.98.3:41805)
I20260929 01:02:17.408607 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db843963cd6e4e8bb352a53772624359" candidate_uuid: "2ac54ced6dad46969c9b046507046048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17fc9c40b7a24115b9d935449d67e1d7"
I20260929 01:02:17.408669 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db843963cd6e4e8bb352a53772624359" candidate_uuid: "2ac54ced6dad46969c9b046507046048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39cdc4e943be4c90a96b54aa8654ca0c"
I20260929 01:02:17.408738 2426 raft_consensus.cc:3037] T db843963cd6e4e8bb352a53772624359 P 17fc9c40b7a24115b9d935449d67e1d7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:17.408813 2567 raft_consensus.cc:3037] T db843963cd6e4e8bb352a53772624359 P 39cdc4e943be4c90a96b54aa8654ca0c [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:17.409580 2426 raft_consensus.cc:2454] T db843963cd6e4e8bb352a53772624359 P 17fc9c40b7a24115b9d935449d67e1d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ac54ced6dad46969c9b046507046048 in term 1.
I20260929 01:02:17.409770 2567 raft_consensus.cc:2454] T db843963cd6e4e8bb352a53772624359 P 39cdc4e943be4c90a96b54aa8654ca0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ac54ced6dad46969c9b046507046048 in term 1.
I20260929 01:02:17.409873 2219 leader_election.cc:302] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17fc9c40b7a24115b9d935449d67e1d7, 2ac54ced6dad46969c9b046507046048; no voters:
I20260929 01:02:17.410045 2716 raft_consensus.cc:2781] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:17.410185 2716 raft_consensus.cc:686] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048 [term 1 LEADER]: Becoming Leader. State: Replica: 2ac54ced6dad46969c9b046507046048, State: Running, Role: LEADER
I20260929 01:02:17.410332 2716 consensus_queue.cc:237] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } }
I20260929 01:02:17.412015 2130 catalog_manager.cc:5495] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ac54ced6dad46969c9b046507046048 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2ac54ced6dad46969c9b046507046048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ac54ced6dad46969c9b046507046048" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 } health_report { overall_health: UNKNOWN } } }
I20260929 01:02:17.440719 2756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:17.436646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46702 (local address 127.0.98.1:45525)
0929 01:02:17.436932 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:17.436940 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:17.436953 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:02:17.436994 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:17.436996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:17.437041 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:17.437123 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:02:17.437321 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:17.438127 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:17.439358 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:17.439705 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:17.439791 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:17.440017 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:02:17.440098 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:02:17.440166 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:17.440527 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 01:02:17.440579 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":113,"threads_started":1}
I20260929 01:02:17.446722 2426 raft_consensus.cc:1261] T db843963cd6e4e8bb352a53772624359 P 17fc9c40b7a24115b9d935449d67e1d7 [term 1 FOLLOWER]: Refusing update from remote peer 2ac54ced6dad46969c9b046507046048: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:17.446740 2567 raft_consensus.cc:1261] T db843963cd6e4e8bb352a53772624359 P 39cdc4e943be4c90a96b54aa8654ca0c [term 1 FOLLOWER]: Refusing update from remote peer 2ac54ced6dad46969c9b046507046048: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:17.447096 2720 consensus_queue.cc:1035] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17fc9c40b7a24115b9d935449d67e1d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:17.447249 2716 consensus_queue.cc:1035] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39cdc4e943be4c90a96b54aa8654ca0c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 01:02:17.461505 2708 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 01:02:19.006726 2130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:19.006914 2130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:19.009145 2130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1344acb25074603a231c2ab0d2ed0e3: Sending DeleteTablet for 3 replicas of tablet db843963cd6e4e8bb352a53772624359
I20260929 01:02:19.009635 2268 tablet_service.cc:1514] Processing DeleteTablet for tablet db843963cd6e4e8bb352a53772624359 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46686
I20260929 01:02:19.009722 2408 tablet_service.cc:1514] Processing DeleteTablet for tablet db843963cd6e4e8bb352a53772624359 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55772
I20260929 01:02:19.009745 2548 tablet_service.cc:1514] Processing DeleteTablet for tablet db843963cd6e4e8bb352a53772624359 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46748
I20260929 01:02:19.009969 2781 tablet_replica.cc:330] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048: stopping tablet replica
I20260929 01:02:19.010099 2781 raft_consensus.cc:2229] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:02:19.010133 2782 tablet_replica.cc:330] T db843963cd6e4e8bb352a53772624359 P 39cdc4e943be4c90a96b54aa8654ca0c: stopping tablet replica
I20260929 01:02:19.010264 2781 raft_consensus.cc:2258] T db843963cd6e4e8bb352a53772624359 P 2ac54ced6dad46969c9b046507046048 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:19.010285 2782 raft_consensus.cc:2229] T db843963cd6e4e8bb352a53772624359 P 39cdc4e943be4c90a96b54aa8654ca0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:19.010391 2782 raft_consensus.cc:2258] T db843963cd6e4e8bb352a53772624359 P 39cdc4e943be4c90a96b54aa8654ca0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:19.010401 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 2203
I20260929 01:02:19.011227 2783 tablet_replica.cc:330] T db843963cd6e4e8bb352a53772624359 P 17fc9c40b7a24115b9d935449d67e1d7: stopping tablet replica
I20260929 01:02:19.011372 2783 raft_consensus.cc:2229] T db843963cd6e4e8bb352a53772624359 P 17fc9c40b7a24115b9d935449d67e1d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:19.011475 2783 raft_consensus.cc:2258] T db843963cd6e4e8bb352a53772624359 P 17fc9c40b7a24115b9d935449d67e1d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:19.012154 2782 ts_tablet_manager.cc:1905] T db843963cd6e4e8bb352a53772624359 P 39cdc4e943be4c90a96b54aa8654ca0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:19.014175 2782 ts_tablet_manager.cc:1918] T db843963cd6e4e8bb352a53772624359 P 39cdc4e943be4c90a96b54aa8654ca0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 01:02:19.014266 2782 log.cc:1196] T db843963cd6e4e8bb352a53772624359 P 39cdc4e943be4c90a96b54aa8654ca0c: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/db843963cd6e4e8bb352a53772624359
I20260929 01:02:19.014503 2782 ts_tablet_manager.cc:1939] T db843963cd6e4e8bb352a53772624359 P 39cdc4e943be4c90a96b54aa8654ca0c: Deleting consensus metadata
I20260929 01:02:19.015208 2115 catalog_manager.cc:4841] TS 39cdc4e943be4c90a96b54aa8654ca0c (127.0.98.3:41805): tablet db843963cd6e4e8bb352a53772624359 (table test-table [id=bfb3bdddd53a48a8951c1c05ba0a2b85]) successfully deleted
I20260929 01:02:19.015282 2783 ts_tablet_manager.cc:1905] T db843963cd6e4e8bb352a53772624359 P 17fc9c40b7a24115b9d935449d67e1d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:19.018599 2783 ts_tablet_manager.cc:1918] T db843963cd6e4e8bb352a53772624359 P 17fc9c40b7a24115b9d935449d67e1d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 01:02:19.018687 2783 log.cc:1196] T db843963cd6e4e8bb352a53772624359 P 17fc9c40b7a24115b9d935449d67e1d7: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790643720886468-392-0/minicluster-data/ts-1/wal/wals/db843963cd6e4e8bb352a53772624359
I20260929 01:02:19.018997 2783 ts_tablet_manager.cc:1939] T db843963cd6e4e8bb352a53772624359 P 17fc9c40b7a24115b9d935449d67e1d7: Deleting consensus metadata
I20260929 01:02:19.019500 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 2343
W20260929 01:02:19.019522 2115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45525 (error 108)
I20260929 01:02:19.021234 2785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:19.020862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45525 (local address 127.0.0.1:46716)
0929 01:02:19.021080 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:19.021154 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":89,"threads_started":1}
W20260929 01:02:19.021553 2115 catalog_manager.cc:4584] TS 2ac54ced6dad46969c9b046507046048 (127.0.98.1:45525): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet db843963cd6e4e8bb352a53772624359: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45525: connect: Connection refused (error 111)
I20260929 01:02:19.025810 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 2483
I20260929 01:02:19.026010 2785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:19.025759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34141 (local address 127.0.0.1:55808)
0929 01:02:19.025873 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:19.025928 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 01:02:19.031947 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 2099
2026-09-29T01:02:19Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6205 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:02:19 dist-test-slave-7zdl krb5kdc[2788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:02:19 dist-test-slave-7zdl krb5kdc[2788](info): set up 2 sockets
Sep 29 01:02:19 dist-test-slave-7zdl krb5kdc[2788](info): commencing operation
krb5kdc: starting...
W20260929 01:02:20.070709 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:02:20 dist-test-slave-7zdl krb5kdc[2788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:02:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:02:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:02:20.187417 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42827
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:20.294524 2801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:20.294806 2801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:20.294906 2801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:20.298811 2801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:02:20.298893 2801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:20.298931 2801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:20.298992 2801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:02:20.299031 2801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:02:20.304216 2801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:20.305241 2801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:20.306229 2801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:02:20.314683 2801 server_base.cc:1034] running on GCE node
W20260929 01:02:20.314694 2809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:20.314824 2807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:20.314996 2806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:20.315546 2801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:20.316610 2801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:20.317852 2801 hybrid_clock.cc:648] HybridClock initialized: now 1790643740317836 us; error 26 us; skew 500 ppm
Sep 29 01:02:20 dist-test-slave-7zdl krb5kdc[2788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:20.320684 2801 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:02:20.321604 2801 webserver.cc:460] Webserver started at http://127.0.98.62:34561/ using document root <none> and password file <none>
I20260929 01:02:20.322160 2801 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:20.322250 2801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:20.322525 2801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:20.324599 2801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "2a03d6c54f9648fdb9f43a77f2f8421b"
format_stamp: "Formatted at 2026-09-29 01:02:20 on dist-test-slave-7zdl"
server_key: "12eeb21d89dc22c1c108b884bedbdff5"
server_key_iv: "fe3fdf00aee13f00468141c7381a3f59"
server_key_version: "encryptionkey@0"
I20260929 01:02:20.325141 2801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "2a03d6c54f9648fdb9f43a77f2f8421b"
format_stamp: "Formatted at 2026-09-29 01:02:20 on dist-test-slave-7zdl"
server_key: "12eeb21d89dc22c1c108b884bedbdff5"
server_key_iv: "fe3fdf00aee13f00468141c7381a3f59"
server_key_version: "encryptionkey@0"
I20260929 01:02:20.328670 2801 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:02:20.330926 2816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:20.332108 2801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:20.332242 2801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "2a03d6c54f9648fdb9f43a77f2f8421b"
format_stamp: "Formatted at 2026-09-29 01:02:20 on dist-test-slave-7zdl"
server_key: "12eeb21d89dc22c1c108b884bedbdff5"
server_key_iv: "fe3fdf00aee13f00468141c7381a3f59"
server_key_version: "encryptionkey@0"
I20260929 01:02:20.332342 2801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:20.349025 2801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:20.352208 2801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:20.352403 2801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:20.359355 2801 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42827
I20260929 01:02:20.359376 2868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42827 every 8 connection(s)
I20260929 01:02:20.360252 2801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:02:20.362836 2869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:20.363155 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 2801
I20260929 01:02:20.363261 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:02:20.363551 392 external_mini_cluster.cc:1428] Setting key 38c49837a3f608ebeb2292ae94f1f5df
I20260929 01:02:20.369797 2869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a03d6c54f9648fdb9f43a77f2f8421b: Bootstrap starting.
I20260929 01:02:20.371968 2869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a03d6c54f9648fdb9f43a77f2f8421b: Neither blocks nor log segments found. Creating new log.
Sep 29 01:02:20 dist-test-slave-7zdl krb5kdc[2788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:20.372730 2869 log.cc:824] T 00000000000000000000000000000000 P 2a03d6c54f9648fdb9f43a77f2f8421b: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:20.374491 2869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a03d6c54f9648fdb9f43a77f2f8421b: No bootstrap required, opened a new log
I20260929 01:02:20.377307 2872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:20.364888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40118 (local address 127.0.98.62:42827)
0929 01:02:20.365335 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:20.365352 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:20.365394 (+ 42us) server_negotiation.cc:407] Connection header received
0929 01:02:20.366171 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:20.366194 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:20.366605 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:20.366975 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:20.367617 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:20.369115 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:20.369976 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:20.370417 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:20.370590 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:20.373014 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:20.373019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:20.373032 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:20.373056 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:20.374734 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:20.375175 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:20.375180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:20.375188 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:20.375297 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:20.375573 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:20.375577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:20.375579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:20.375944 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:20.376148 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:20.376378 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 01:02:20.376648 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":100,"threads_started":1}
I20260929 01:02:20.378028 2869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a03d6c54f9648fdb9f43a77f2f8421b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a03d6c54f9648fdb9f43a77f2f8421b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42827 } }
I20260929 01:02:20.378275 2869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a03d6c54f9648fdb9f43a77f2f8421b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:20.378337 2869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a03d6c54f9648fdb9f43a77f2f8421b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a03d6c54f9648fdb9f43a77f2f8421b, State: Initialized, Role: FOLLOWER
I20260929 01:02:20.378767 2869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a03d6c54f9648fdb9f43a77f2f8421b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a03d6c54f9648fdb9f43a77f2f8421b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42827 } }
I20260929 01:02:20.378938 2869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a03d6c54f9648fdb9f43a77f2f8421b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:02:20.379000 2869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a03d6c54f9648fdb9f43a77f2f8421b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:02:20.379101 2869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a03d6c54f9648fdb9f43a77f2f8421b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:20.379993 2869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a03d6c54f9648fdb9f43a77f2f8421b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a03d6c54f9648fdb9f43a77f2f8421b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42827 } }
I20260929 01:02:20.380276 2869 leader_election.cc:302] T 00000000000000000000000000000000 P 2a03d6c54f9648fdb9f43a77f2f8421b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a03d6c54f9648fdb9f43a77f2f8421b; no voters:
I20260929 01:02:20.380498 2869 leader_election.cc:288] T 00000000000000000000000000000000 P 2a03d6c54f9648fdb9f43a77f2f8421b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:02:20.380609 2874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a03d6c54f9648fdb9f43a77f2f8421b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:20.380841 2874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a03d6c54f9648fdb9f43a77f2f8421b [term 1 LEADER]: Becoming Leader. State: Replica: 2a03d6c54f9648fdb9f43a77f2f8421b, State: Running, Role: LEADER
I20260929 01:02:20.381120 2874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a03d6c54f9648fdb9f43a77f2f8421b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a03d6c54f9648fdb9f43a77f2f8421b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42827 } }
I20260929 01:02:20.381669 2869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a03d6c54f9648fdb9f43a77f2f8421b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:02:20.382720 2875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a03d6c54f9648fdb9f43a77f2f8421b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a03d6c54f9648fdb9f43a77f2f8421b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a03d6c54f9648fdb9f43a77f2f8421b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42827 } } }
I20260929 01:02:20.382876 2875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a03d6c54f9648fdb9f43a77f2f8421b [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:20.383134 2876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a03d6c54f9648fdb9f43a77f2f8421b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a03d6c54f9648fdb9f43a77f2f8421b. Latest consensus state: current_term: 1 leader_uuid: "2a03d6c54f9648fdb9f43a77f2f8421b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a03d6c54f9648fdb9f43a77f2f8421b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42827 } } }
I20260929 01:02:20.383221 2876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a03d6c54f9648fdb9f43a77f2f8421b [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:20.383663 2883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:02:20.386520 2883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:02:20.391523 2883 catalog_manager.cc:1347] Generated new cluster ID: feecc9ab4b2245f087c3088c7af96883
I20260929 01:02:20.391582 2883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:02:20.411266 2883 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:02:20.412156 2883 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:02:20.421198 2883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a03d6c54f9648fdb9f43a77f2f8421b: Generated new TSK 0
I20260929 01:02:20.421850 2883 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 01:02:20.453675 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42827
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:20.575027 2895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:20.575265 2895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:20.575306 2895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:20.579356 2895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:20.579403 2895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:20.579502 2895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:02:20.584995 2895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:20.586004 2895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:20.587037 2895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:20.596444 2901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:20.596557 2895 server_base.cc:1034] running on GCE node
W20260929 01:02:20.596585 2903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:20.596669 2900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:20.597162 2895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:20.597774 2895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:20.598956 2895 hybrid_clock.cc:648] HybridClock initialized: now 1790643740598940 us; error 27 us; skew 500 ppm
Sep 29 01:02:20 dist-test-slave-7zdl krb5kdc[2788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:20.601642 2895 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:02:20.602458 2895 webserver.cc:460] Webserver started at http://127.0.98.1:41667/ using document root <none> and password file <none>
I20260929 01:02:20.603142 2895 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:20.603223 2895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:20.603451 2895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:20.605772 2895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "9fcc7cc6d62a44668fec93c53a9fcdb8"
format_stamp: "Formatted at 2026-09-29 01:02:20 on dist-test-slave-7zdl"
server_key: "5ad12be1a5f16d808dc04dd2dfbac50e"
server_key_iv: "0fcdb6c90ca4e20539a247e53467ff30"
server_key_version: "encryptionkey@0"
I20260929 01:02:20.606441 2895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9fcc7cc6d62a44668fec93c53a9fcdb8"
format_stamp: "Formatted at 2026-09-29 01:02:20 on dist-test-slave-7zdl"
server_key: "5ad12be1a5f16d808dc04dd2dfbac50e"
server_key_iv: "0fcdb6c90ca4e20539a247e53467ff30"
server_key_version: "encryptionkey@0"
I20260929 01:02:20.609900 2895 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 01:02:20.612164 2910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:20.613307 2895 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:02:20.613442 2895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "9fcc7cc6d62a44668fec93c53a9fcdb8"
format_stamp: "Formatted at 2026-09-29 01:02:20 on dist-test-slave-7zdl"
server_key: "5ad12be1a5f16d808dc04dd2dfbac50e"
server_key_iv: "0fcdb6c90ca4e20539a247e53467ff30"
server_key_version: "encryptionkey@0"
I20260929 01:02:20.613546 2895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:20.638411 2895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:20.641749 2895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:20.641916 2895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:20.642436 2895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:20.643256 2895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:20.643302 2895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:20.643339 2895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:20.643355 2895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:20.651546 2895 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35603
I20260929 01:02:20.651636 3023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35603 every 8 connection(s)
I20260929 01:02:20.652505 2895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:02:20.660207 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 2895
I20260929 01:02:20.660334 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:02:20.660601 392 external_mini_cluster.cc:1428] Setting key 70fb01cb8fdb47aaa7ea67f8f590ef24
Sep 29 01:02:20 dist-test-slave-7zdl krb5kdc[2788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:20.666730 2872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:20.654279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48733 (local address 127.0.98.62:42827)
0929 01:02:20.654426 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:20.654430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:20.655415 (+ 985us) server_negotiation.cc:407] Connection header received
0929 01:02:20.656734 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:20.656737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:20.656795 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:20.656868 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:20.657943 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:20.658450 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:20.659643 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:20.659874 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:20.660032 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:20.662518 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:20.662521 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:20.662523 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:20.662545 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:20.664304 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:20.664885 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:20.664888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:20.664889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:20.664996 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:20.665482 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:20.665488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:20.665490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:20.665758 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:20.665863 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:20.666502 (+ 639us) server_negotiation.cc:299] Negotiation successful
0929 01:02:20.666611 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 01:02:20.667977 3026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:20.654705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42827 (local address 127.0.98.1:48733)
0929 01:02:20.655269 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:20.655309 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:20.656553 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:20.657027 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:20.657033 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:20.657377 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:20.657823 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:20.657835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:20.658560 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:20.658562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:20.659515 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:20.659524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:20.660078 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:20.660081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:20.660217 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:20.660858 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:20.660873 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:20.662355 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:20.664509 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:20.664514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:20.664525 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:20.664765 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:20.665089 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:20.665095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:20.665100 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:20.665278 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:20.665873 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:20.665882 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:20.666204 (+ 322us) client_negotiation.cc:769] Sending connection context
0929 01:02:20.666452 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 01:02:20.666828 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":108,"threads_started":1}
I20260929 01:02:20.669486 3024 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42827
I20260929 01:02:20.669740 3024 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 01:02:20.670250 3024 heartbeater.cc:511] Master 127.0.98.62:42827 requested a full tablet report, sending...
I20260929 01:02:20.671810 2833 ts_manager.cc:194] Registered new tserver with Master: 9fcc7cc6d62a44668fec93c53a9fcdb8 (127.0.98.1:35603)
I20260929 01:02:20.672906 2833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 01:02:20.689745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42827
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:20.796291 3029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:20.796514 3029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:20.796552 3029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:20.800432 3029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:20.800489 3029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:20.800554 3029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:02:20.805550 3029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:20.806457 3029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:20.807452 3029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:20.816905 3034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:20.816920 3037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:20.816948 3035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:20.817157 3029 server_base.cc:1034] running on GCE node
I20260929 01:02:20.817593 3029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:20.818256 3029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:20.819404 3029 hybrid_clock.cc:648] HybridClock initialized: now 1790643740819379 us; error 53 us; skew 500 ppm
Sep 29 01:02:20 dist-test-slave-7zdl krb5kdc[2788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:20.821923 3029 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:02:20.822723 3029 webserver.cc:460] Webserver started at http://127.0.98.2:41903/ using document root <none> and password file <none>
I20260929 01:02:20.823294 3029 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:20.823391 3029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:20.823613 3029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:20.825539 3029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "58685d14fdac4153b24156311c8901b2"
format_stamp: "Formatted at 2026-09-29 01:02:20 on dist-test-slave-7zdl"
server_key: "6a4ee0321e0a33313a4ddb521921bebc"
server_key_iv: "224f8bb96a347cbdd609e8f92ed5c1d7"
server_key_version: "encryptionkey@0"
I20260929 01:02:20.826119 3029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "58685d14fdac4153b24156311c8901b2"
format_stamp: "Formatted at 2026-09-29 01:02:20 on dist-test-slave-7zdl"
server_key: "6a4ee0321e0a33313a4ddb521921bebc"
server_key_iv: "224f8bb96a347cbdd609e8f92ed5c1d7"
server_key_version: "encryptionkey@0"
I20260929 01:02:20.829571 3029 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 01:02:20.831916 3044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:20.833068 3029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:02:20.833200 3029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "58685d14fdac4153b24156311c8901b2"
format_stamp: "Formatted at 2026-09-29 01:02:20 on dist-test-slave-7zdl"
server_key: "6a4ee0321e0a33313a4ddb521921bebc"
server_key_iv: "224f8bb96a347cbdd609e8f92ed5c1d7"
server_key_version: "encryptionkey@0"
I20260929 01:02:20.833302 3029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:20.860718 3029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:20.863624 3029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:20.863827 3029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:20.864421 3029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:20.865201 3029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:20.865281 3029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:20.865404 3029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:20.865454 3029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:20.874423 3029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35505
I20260929 01:02:20.874445 3157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35505 every 8 connection(s)
I20260929 01:02:20.875351 3029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:02:20.875641 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 3029
I20260929 01:02:20.875769 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:02:20.876118 392 external_mini_cluster.cc:1428] Setting key 4064ca183420191b1067f178330b9496
Sep 29 01:02:20 dist-test-slave-7zdl krb5kdc[2788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 01:02:20.891106 2872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:20.877173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50521 (local address 127.0.98.62:42827)
0929 01:02:20.877308 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:20.877313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:20.878271 (+ 958us) server_negotiation.cc:407] Connection header received
0929 01:02:20.880042 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:20.880048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:20.880103 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:20.880197 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:20.881366 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:20.881975 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:20.883931 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:20.884242 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:20.884328 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:20.887120 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:20.887124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:20.887126 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:20.887147 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:20.888747 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:20.889246 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:20.889255 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:20.889257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:20.889344 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:20.889727 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:20.889729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:20.889731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:20.889908 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:20.890020 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:20.890703 (+ 683us) server_negotiation.cc:299] Negotiation successful
0929 01:02:20.890861 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 01:02:20.891932 3160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:20.877629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42827 (local address 127.0.98.2:50521)
0929 01:02:20.878105 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:20.878161 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:20.879845 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:20.880418 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:20.880425 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:20.880768 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:20.881227 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:20.881240 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:20.882102 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:20.882108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:20.883777 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:20.883792 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:20.884402 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:20.884408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:20.884648 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:20.885278 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:20.885299 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:20.886927 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:20.888863 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:20.888869 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:20.888883 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:20.889129 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:20.889495 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:20.889498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:20.889499 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:20.889629 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:20.890020 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:20.890025 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:20.890417 (+ 392us) client_negotiation.cc:769] Sending connection context
0929 01:02:20.890603 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 01:02:20.890914 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":109,"threads_started":1}
I20260929 01:02:20.893206 3158 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42827
I20260929 01:02:20.893450 3158 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:20.893916 3158 heartbeater.cc:511] Master 127.0.98.62:42827 requested a full tablet report, sending...
I20260929 01:02:20.895027 2833 ts_manager.cc:194] Registered new tserver with Master: 58685d14fdac4153b24156311c8901b2 (127.0.98.2:35505)
I20260929 01:02:20.895581 2833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 01:02:20.907263 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42827
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:21.020056 3163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:21.020292 3163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:21.020341 3163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:21.024652 3163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:21.024708 3163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:21.024876 3163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:02:21.030217 3163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:21.031257 3163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:21.032280 3163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:21.042138 3171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:21.042140 3169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:21.042155 3168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:21.042250 3163 server_base.cc:1034] running on GCE node
I20260929 01:02:21.042858 3163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:21.043454 3163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:21.044610 3163 hybrid_clock.cc:648] HybridClock initialized: now 1790643741044597 us; error 24 us; skew 500 ppm
Sep 29 01:02:21 dist-test-slave-7zdl krb5kdc[2788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:21.047145 3163 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:02:21.047966 3163 webserver.cc:460] Webserver started at http://127.0.98.3:34043/ using document root <none> and password file <none>
I20260929 01:02:21.048494 3163 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:21.048561 3163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:21.048786 3163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:21.050705 3163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "c8ed5269499c4d4ab4e3ac46db278991"
format_stamp: "Formatted at 2026-09-29 01:02:21 on dist-test-slave-7zdl"
server_key: "b5cb4f0bed9db5afe47eabfd29d470c6"
server_key_iv: "6165bf4fe8f2b3c22efc6e6a05ccc3a9"
server_key_version: "encryptionkey@0"
I20260929 01:02:21.051301 3163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c8ed5269499c4d4ab4e3ac46db278991"
format_stamp: "Formatted at 2026-09-29 01:02:21 on dist-test-slave-7zdl"
server_key: "b5cb4f0bed9db5afe47eabfd29d470c6"
server_key_iv: "6165bf4fe8f2b3c22efc6e6a05ccc3a9"
server_key_version: "encryptionkey@0"
I20260929 01:02:21.054788 3163 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:02:21.057070 3178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:21.058202 3163 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:21.058312 3163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "c8ed5269499c4d4ab4e3ac46db278991"
format_stamp: "Formatted at 2026-09-29 01:02:21 on dist-test-slave-7zdl"
server_key: "b5cb4f0bed9db5afe47eabfd29d470c6"
server_key_iv: "6165bf4fe8f2b3c22efc6e6a05ccc3a9"
server_key_version: "encryptionkey@0"
I20260929 01:02:21.058401 3163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:21.077934 3163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:21.080627 3163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:21.080806 3163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:21.081391 3163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:21.082223 3163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:21.082289 3163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:21.082351 3163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:21.082379 3163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:21.090762 3163 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43505
I20260929 01:02:21.090786 3291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43505 every 8 connection(s)
I20260929 01:02:21.091634 3163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:02:21.093279 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 3163
I20260929 01:02:21.093411 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:02:21.093698 392 external_mini_cluster.cc:1428] Setting key 9fe16521c7b79f85ce5481d703fe5aec
Sep 29 01:02:21 dist-test-slave-7zdl krb5kdc[2788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:21.104008 2872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:21.093323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43207 (local address 127.0.98.62:42827)
0929 01:02:21.093487 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:21.093493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:21.094414 (+ 921us) server_negotiation.cc:407] Connection header received
0929 01:02:21.095799 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:21.095801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:21.095843 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:21.095907 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:21.096907 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:21.097410 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:21.098463 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:21.098668 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:21.098740 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:21.101063 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:21.101066 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:21.101067 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:21.101089 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:21.102372 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:21.102792 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:21.102822 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:21.102824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:21.102897 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:21.103167 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:21.103169 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:21.103170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:21.103326 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:21.103398 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:21.103791 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 01:02:21.103890 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 01:02:21.104803 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:21.093805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42827 (local address 127.0.98.3:43207)
0929 01:02:21.094272 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:21.094321 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:21.095635 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:21.096035 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:21.096042 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:21.096362 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:21.096798 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:21.096810 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:21.097511 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:21.097514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:21.098368 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:21.098376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:21.098780 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:21.098782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:21.098978 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:21.099551 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:21.099566 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:21.100910 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:21.102470 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:21.102474 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:21.102484 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:21.102715 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:21.102973 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:21.102975 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:21.102976 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:21.103095 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:21.103402 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:21.103407 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:21.103607 (+ 200us) client_negotiation.cc:769] Sending connection context
0929 01:02:21.103750 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 01:02:21.103988 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":131,"threads_started":1}
I20260929 01:02:21.105882 3292 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42827
I20260929 01:02:21.106112 3292 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:21.106643 3292 heartbeater.cc:511] Master 127.0.98.62:42827 requested a full tablet report, sending...
I20260929 01:02:21.107669 2833 ts_manager.cc:194] Registered new tserver with Master: c8ed5269499c4d4ab4e3ac46db278991 (127.0.98.3:43505)
I20260929 01:02:21.108175 2833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43207
I20260929 01:02:21.119030 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 01:02:21.131629 2872 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:21.130238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40120 (local address 127.0.98.62:42827)
0929 01:02:21.130407 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:21.130411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:21.130477 (+ 66us) server_negotiation.cc:407] Connection header received
0929 01:02:21.130586 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:21.130589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:21.130630 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:21.130700 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:21.131468 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40120: BlockingRecv error: recv got EOF from 127.0.0.1:40120 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 01:02:21.133038 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 2895
I20260929 01:02:21.138726 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 3029
I20260929 01:02:21.143900 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 3163
I20260929 01:02:21.148810 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 2801
2026-09-29T01:02:21Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2115 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:02:21 dist-test-slave-7zdl krb5kdc[3304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:02:21 dist-test-slave-7zdl krb5kdc[3304](info): set up 2 sockets
Sep 29 01:02:21 dist-test-slave-7zdl krb5kdc[3304](info): commencing operation
krb5kdc: starting...
W20260929 01:02:22.182727 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:02:22 dist-test-slave-7zdl krb5kdc[3304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:02:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:02:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:02:22.295958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34871
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:22.401438 3317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:22.401744 3317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:22.401819 3317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:22.405833 3317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:02:22.405926 3317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:22.405963 3317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:22.405999 3317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:02:22.406035 3317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:02:22.411193 3317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:22.412233 3317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:22.413250 3317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:22.421608 3322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:22.421674 3323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:22.421737 3317 server_base.cc:1034] running on GCE node
W20260929 01:02:22.421612 3325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:22.422427 3317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:22.423519 3317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:22.424698 3317 hybrid_clock.cc:648] HybridClock initialized: now 1790643742424680 us; error 31 us; skew 500 ppm
Sep 29 01:02:22 dist-test-slave-7zdl krb5kdc[3304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:22.431375 3317 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:02:22.432266 3317 webserver.cc:460] Webserver started at http://127.0.98.62:43421/ using document root <none> and password file <none>
I20260929 01:02:22.432817 3317 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:22.432905 3317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:22.433137 3317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:22.435180 3317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "3d2cefb99e494fe18300ee0a765556e0"
format_stamp: "Formatted at 2026-09-29 01:02:22 on dist-test-slave-7zdl"
server_key: "031a454cf3731507c5c22048feebc424"
server_key_iv: "6847d60a0695365e8e4b8db0bdaaef90"
server_key_version: "encryptionkey@0"
I20260929 01:02:22.435737 3317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "3d2cefb99e494fe18300ee0a765556e0"
format_stamp: "Formatted at 2026-09-29 01:02:22 on dist-test-slave-7zdl"
server_key: "031a454cf3731507c5c22048feebc424"
server_key_iv: "6847d60a0695365e8e4b8db0bdaaef90"
server_key_version: "encryptionkey@0"
I20260929 01:02:22.439332 3317 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:02:22.441627 3332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:22.442909 3317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:02:22.443040 3317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "3d2cefb99e494fe18300ee0a765556e0"
format_stamp: "Formatted at 2026-09-29 01:02:22 on dist-test-slave-7zdl"
server_key: "031a454cf3731507c5c22048feebc424"
server_key_iv: "6847d60a0695365e8e4b8db0bdaaef90"
server_key_version: "encryptionkey@0"
I20260929 01:02:22.443145 3317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:22.457702 3317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:22.460378 3317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:22.460574 3317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:22.467159 3317 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34871
I20260929 01:02:22.467418 3384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34871 every 8 connection(s)
I20260929 01:02:22.468114 3317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:02:22.470847 3385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:22.471276 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 3317
I20260929 01:02:22.471367 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:02:22.471654 392 external_mini_cluster.cc:1428] Setting key 29306f66d9593f2defe80a62d4c1ee0e
I20260929 01:02:22.477375 3385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d2cefb99e494fe18300ee0a765556e0: Bootstrap starting.
I20260929 01:02:22.480065 3385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d2cefb99e494fe18300ee0a765556e0: Neither blocks nor log segments found. Creating new log.
Sep 29 01:02:22 dist-test-slave-7zdl krb5kdc[3304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:22.480765 3385 log.cc:824] T 00000000000000000000000000000000 P 3d2cefb99e494fe18300ee0a765556e0: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:22.482578 3385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d2cefb99e494fe18300ee0a765556e0: No bootstrap required, opened a new log
I20260929 01:02:22.485198 3388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:22.472819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58376 (local address 127.0.98.62:34871)
0929 01:02:22.473209 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:22.473224 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:22.473257 (+ 33us) server_negotiation.cc:407] Connection header received
0929 01:02:22.473895 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:22.473930 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:22.474238 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:22.474510 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:22.475426 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:22.476355 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:22.477506 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:22.477970 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:22.478094 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:22.480738 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:22.480743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:22.480755 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:22.480779 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:22.482514 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:22.483029 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:22.483035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:22.483044 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:22.483148 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:22.483420 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:22.483423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:22.483424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:22.483795 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:22.483994 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:22.484283 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 01:02:22.484549 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":132,"threads_started":1}
I20260929 01:02:22.486179 3385 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d2cefb99e494fe18300ee0a765556e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2cefb99e494fe18300ee0a765556e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34871 } }
I20260929 01:02:22.486433 3385 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d2cefb99e494fe18300ee0a765556e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:22.486492 3385 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d2cefb99e494fe18300ee0a765556e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d2cefb99e494fe18300ee0a765556e0, State: Initialized, Role: FOLLOWER
I20260929 01:02:22.486964 3385 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d2cefb99e494fe18300ee0a765556e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2cefb99e494fe18300ee0a765556e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34871 } }
I20260929 01:02:22.487134 3385 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d2cefb99e494fe18300ee0a765556e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:02:22.487215 3385 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d2cefb99e494fe18300ee0a765556e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:02:22.487313 3385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d2cefb99e494fe18300ee0a765556e0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:22.487882 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:02:22.488307 3385 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d2cefb99e494fe18300ee0a765556e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2cefb99e494fe18300ee0a765556e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34871 } }
I20260929 01:02:22.488629 3385 leader_election.cc:302] T 00000000000000000000000000000000 P 3d2cefb99e494fe18300ee0a765556e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d2cefb99e494fe18300ee0a765556e0; no voters:
I20260929 01:02:22.488898 3385 leader_election.cc:288] T 00000000000000000000000000000000 P 3d2cefb99e494fe18300ee0a765556e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:02:22.488999 3390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d2cefb99e494fe18300ee0a765556e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:22.489215 3390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d2cefb99e494fe18300ee0a765556e0 [term 1 LEADER]: Becoming Leader. State: Replica: 3d2cefb99e494fe18300ee0a765556e0, State: Running, Role: LEADER
I20260929 01:02:22.489598 3390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d2cefb99e494fe18300ee0a765556e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2cefb99e494fe18300ee0a765556e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34871 } }
I20260929 01:02:22.490129 3385 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d2cefb99e494fe18300ee0a765556e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:02:22.491465 3391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d2cefb99e494fe18300ee0a765556e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d2cefb99e494fe18300ee0a765556e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2cefb99e494fe18300ee0a765556e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34871 } } }
I20260929 01:02:22.491603 3391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d2cefb99e494fe18300ee0a765556e0 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:22.491464 3392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d2cefb99e494fe18300ee0a765556e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d2cefb99e494fe18300ee0a765556e0. Latest consensus state: current_term: 1 leader_uuid: "3d2cefb99e494fe18300ee0a765556e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d2cefb99e494fe18300ee0a765556e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34871 } } }
I20260929 01:02:22.491882 3392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d2cefb99e494fe18300ee0a765556e0 [sys.catalog]: This master's current role is: LEADER
W20260929 01:02:22.494572 3406 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d2cefb99e494fe18300ee0a765556e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:02:22.494657 3406 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:02:22.494771 3402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:02:22.495621 3402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:02:22.500773 3402 catalog_manager.cc:1347] Generated new cluster ID: 4701063b98154589b46cdf1cf2dfcd34
I20260929 01:02:22.500831 3402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:02:22.511602 3388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:22.506213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58390 (local address 127.0.98.62:34871)
0929 01:02:22.506317 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:22.506321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:22.506594 (+ 273us) server_negotiation.cc:407] Connection header received
0929 01:02:22.506713 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:22.506716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:22.506756 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:22.506862 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:22.507438 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:22.508208 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:22.509344 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:22.509684 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:22.509766 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:22.510658 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:22.510662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:22.510664 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:02:22.511181 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:22.511209 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:02:22.511226 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:22.511304 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:22.511415 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 01:02:22.511473 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 01:02:22.518792 3402 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:02:22.519976 3402 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:02:22.527078 3402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d2cefb99e494fe18300ee0a765556e0: Generated new TSK 0
I20260929 01:02:22.527724 3402 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:02:22.536311 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 3317
2026-09-29T01:02:22Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1385 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:02:22 dist-test-slave-7zdl krb5kdc[3418](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:02:22 dist-test-slave-7zdl krb5kdc[3418](info): set up 2 sockets
Sep 29 01:02:22 dist-test-slave-7zdl krb5kdc[3418](info): commencing operation
krb5kdc: starting...
W20260929 01:02:23.598878 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:02:23 dist-test-slave-7zdl krb5kdc[3418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:02:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:02:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:02:23.718310 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39671
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:23.825378 3431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:23.825651 3431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:23.825729 3431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:23.829778 3431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:02:23.829856 3431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:23.829895 3431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:23.829946 3431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:02:23.829982 3431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:02:23.835098 3431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:23.836057 3431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:23.837033 3431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:23.845223 3437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:23.845187 3436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:23.845187 3439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:23.845300 3431 server_base.cc:1034] running on GCE node
I20260929 01:02:23.846021 3431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:23.847115 3431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:23.848277 3431 hybrid_clock.cc:648] HybridClock initialized: now 1790643743848245 us; error 43 us; skew 500 ppm
Sep 29 01:02:23 dist-test-slave-7zdl krb5kdc[3418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:23.851015 3431 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:02:23.851869 3431 webserver.cc:460] Webserver started at http://127.0.98.62:38969/ using document root <none> and password file <none>
I20260929 01:02:23.852399 3431 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:23.852483 3431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:23.852694 3431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:23.854606 3431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "fbd9d895732d487a8fbd5716fe5ed03d"
format_stamp: "Formatted at 2026-09-29 01:02:23 on dist-test-slave-7zdl"
server_key: "42c4eaddf0843c1ef4428dbcb11d90fa"
server_key_iv: "3d76c8cb4cc9c68ec1b1589a35b8b62f"
server_key_version: "encryptionkey@0"
I20260929 01:02:23.855230 3431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "fbd9d895732d487a8fbd5716fe5ed03d"
format_stamp: "Formatted at 2026-09-29 01:02:23 on dist-test-slave-7zdl"
server_key: "42c4eaddf0843c1ef4428dbcb11d90fa"
server_key_iv: "3d76c8cb4cc9c68ec1b1589a35b8b62f"
server_key_version: "encryptionkey@0"
I20260929 01:02:23.858791 3431 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:02:23.861089 3446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:23.862303 3431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:23.862514 3431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "fbd9d895732d487a8fbd5716fe5ed03d"
format_stamp: "Formatted at 2026-09-29 01:02:23 on dist-test-slave-7zdl"
server_key: "42c4eaddf0843c1ef4428dbcb11d90fa"
server_key_iv: "3d76c8cb4cc9c68ec1b1589a35b8b62f"
server_key_version: "encryptionkey@0"
I20260929 01:02:23.862622 3431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:23.872970 3431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:23.876124 3431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:23.876330 3431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:23.883199 3431 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39671
I20260929 01:02:23.883469 3498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39671 every 8 connection(s)
I20260929 01:02:23.884179 3431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:02:23.886838 3499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:23.892164 3499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbd9d895732d487a8fbd5716fe5ed03d: Bootstrap starting.
I20260929 01:02:23.893827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 3431
I20260929 01:02:23.893965 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:02:23.894233 392 external_mini_cluster.cc:1428] Setting key 68eec0f7daae1634de68a7969b37bad0
I20260929 01:02:23.894662 3499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbd9d895732d487a8fbd5716fe5ed03d: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:23.895524 3499 log.cc:824] T 00000000000000000000000000000000 P fbd9d895732d487a8fbd5716fe5ed03d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:23.897384 3499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbd9d895732d487a8fbd5716fe5ed03d: No bootstrap required, opened a new log
I20260929 01:02:23.901472 3499 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbd9d895732d487a8fbd5716fe5ed03d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbd9d895732d487a8fbd5716fe5ed03d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39671 } }
I20260929 01:02:23.901770 3499 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbd9d895732d487a8fbd5716fe5ed03d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:23.901830 3499 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbd9d895732d487a8fbd5716fe5ed03d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbd9d895732d487a8fbd5716fe5ed03d, State: Initialized, Role: FOLLOWER
I20260929 01:02:23.902367 3499 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbd9d895732d487a8fbd5716fe5ed03d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbd9d895732d487a8fbd5716fe5ed03d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39671 } }
I20260929 01:02:23.902532 3499 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbd9d895732d487a8fbd5716fe5ed03d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:02:23.902606 3499 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbd9d895732d487a8fbd5716fe5ed03d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:02:23.902721 3499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbd9d895732d487a8fbd5716fe5ed03d [term 0 FOLLOWER]: Advancing to term 1
Sep 29 01:02:23 dist-test-slave-7zdl krb5kdc[3418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:23.903644 3499 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbd9d895732d487a8fbd5716fe5ed03d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbd9d895732d487a8fbd5716fe5ed03d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39671 } }
I20260929 01:02:23.904042 3499 leader_election.cc:302] T 00000000000000000000000000000000 P fbd9d895732d487a8fbd5716fe5ed03d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbd9d895732d487a8fbd5716fe5ed03d; no voters:
I20260929 01:02:23.904417 3499 leader_election.cc:288] T 00000000000000000000000000000000 P fbd9d895732d487a8fbd5716fe5ed03d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:02:23.904523 3504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbd9d895732d487a8fbd5716fe5ed03d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:23.904767 3504 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbd9d895732d487a8fbd5716fe5ed03d [term 1 LEADER]: Becoming Leader. State: Replica: fbd9d895732d487a8fbd5716fe5ed03d, State: Running, Role: LEADER
I20260929 01:02:23.905115 3504 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbd9d895732d487a8fbd5716fe5ed03d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbd9d895732d487a8fbd5716fe5ed03d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39671 } }
I20260929 01:02:23.905730 3499 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbd9d895732d487a8fbd5716fe5ed03d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:02:23.906534 3506 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbd9d895732d487a8fbd5716fe5ed03d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbd9d895732d487a8fbd5716fe5ed03d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbd9d895732d487a8fbd5716fe5ed03d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39671 } } }
I20260929 01:02:23.906666 3506 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbd9d895732d487a8fbd5716fe5ed03d [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:23.906917 3505 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbd9d895732d487a8fbd5716fe5ed03d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbd9d895732d487a8fbd5716fe5ed03d. Latest consensus state: current_term: 1 leader_uuid: "fbd9d895732d487a8fbd5716fe5ed03d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbd9d895732d487a8fbd5716fe5ed03d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39671 } } }
I20260929 01:02:23.907092 3505 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbd9d895732d487a8fbd5716fe5ed03d [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:23.909641 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:23.895409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43936 (local address 127.0.98.62:39671)
0929 01:02:23.895866 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:23.895884 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:23.895922 (+ 38us) server_negotiation.cc:407] Connection header received
0929 01:02:23.896527 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:23.896544 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:23.896852 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:23.897126 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:23.898019 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:23.899066 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:23.900371 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:23.900869 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:23.900996 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:23.903766 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:23.903773 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:23.903788 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:23.903817 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:23.905847 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:23.906404 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:23.906410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:23.906415 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:23.906565 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:23.906958 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:23.906961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:23.906963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:23.907483 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:23.907673 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:23.908468 (+ 795us) server_negotiation.cc:299] Negotiation successful
0929 01:02:23.908747 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":157,"threads_started":1}
W20260929 01:02:23.910912 3519 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fbd9d895732d487a8fbd5716fe5ed03d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:02:23.910997 3519 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:02:23.911089 3513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:02:23.911741 3513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:02:23.916885 3513 catalog_manager.cc:1347] Generated new cluster ID: 0ca2a4235c7d47108f7504baa20bc871
I20260929 01:02:23.916949 3513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:02:23.930361 3513 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:02:23.931308 3513 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:02:23.943497 3513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbd9d895732d487a8fbd5716fe5ed03d: Generated new TSK 0
I20260929 01:02:23.944150 3513 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 01:02:23.971882 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39671
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:24.084007 3525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:24.084311 3525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:24.084396 3525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:24.088418 3525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:24.088496 3525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:24.088588 3525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:02:24.093600 3525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:24.094637 3525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:24.095656 3525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:24.105129 3533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:24.105209 3530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:24.105140 3531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:24.105284 3525 server_base.cc:1034] running on GCE node
I20260929 01:02:24.105798 3525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:24.106442 3525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:24.107589 3525 hybrid_clock.cc:648] HybridClock initialized: now 1790643744107561 us; error 41 us; skew 500 ppm
Sep 29 01:02:24 dist-test-slave-7zdl krb5kdc[3418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:24.110216 3525 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:02:24.111099 3525 webserver.cc:460] Webserver started at http://127.0.98.1:42741/ using document root <none> and password file <none>
I20260929 01:02:24.111676 3525 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:24.111737 3525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:24.111937 3525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:24.113888 3525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "40fd388b4cce46849bb9d16ac46d02eb"
format_stamp: "Formatted at 2026-09-29 01:02:24 on dist-test-slave-7zdl"
server_key: "7ae8e592dfa50533c2f374307c772fdf"
server_key_iv: "e620575609f509734c0a92ed6f6f9027"
server_key_version: "encryptionkey@0"
I20260929 01:02:24.114427 3525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "40fd388b4cce46849bb9d16ac46d02eb"
format_stamp: "Formatted at 2026-09-29 01:02:24 on dist-test-slave-7zdl"
server_key: "7ae8e592dfa50533c2f374307c772fdf"
server_key_iv: "e620575609f509734c0a92ed6f6f9027"
server_key_version: "encryptionkey@0"
I20260929 01:02:24.118005 3525 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 01:02:24.120321 3540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:24.121531 3525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:02:24.121622 3525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "40fd388b4cce46849bb9d16ac46d02eb"
format_stamp: "Formatted at 2026-09-29 01:02:24 on dist-test-slave-7zdl"
server_key: "7ae8e592dfa50533c2f374307c772fdf"
server_key_iv: "e620575609f509734c0a92ed6f6f9027"
server_key_version: "encryptionkey@0"
I20260929 01:02:24.121703 3525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:24.135840 3525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:24.138445 3525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:24.138595 3525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:24.139096 3525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:24.139827 3525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:24.139873 3525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:24.139909 3525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:24.139930 3525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:24.148581 3525 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36407
I20260929 01:02:24.148604 3653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36407 every 8 connection(s)
I20260929 01:02:24.149444 3525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:02:24.158144 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 3525
I20260929 01:02:24.158236 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
Sep 29 01:02:24 dist-test-slave-7zdl krb5kdc[3418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:24.158552 392 external_mini_cluster.cc:1428] Setting key 50c2cfb8f58f2f19e8d95e1a565d05f5
I20260929 01:02:24.163115 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:24.151174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45155 (local address 127.0.98.62:39671)
0929 01:02:24.151293 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:24.151297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:24.152132 (+ 835us) server_negotiation.cc:407] Connection header received
0929 01:02:24.153438 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:24.153441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:24.153491 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:24.153565 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:24.154638 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:24.155237 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:24.156352 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:24.156572 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:24.156646 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:24.159104 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:24.159111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:24.159115 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:24.159145 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:24.160781 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:24.161392 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:24.161394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:24.161396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:24.161467 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:24.161861 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:24.161866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:24.161869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:24.162130 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:24.162237 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:24.162692 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 01:02:24.162854 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 01:02:24.163833 3656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:24.151569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39671 (local address 127.0.98.1:45155)
0929 01:02:24.151990 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:24.152031 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:24.153244 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:24.153693 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:24.153699 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:24.154056 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:24.154521 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:24.154534 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:24.155335 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:24.155337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:24.156253 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:24.156262 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:24.156688 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:24.156691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:24.156820 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:24.157320 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:24.157335 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:24.158885 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:24.160895 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:24.160903 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:24.160920 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:24.161298 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:24.161553 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:24.161559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:24.161563 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:24.161756 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:24.162231 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:24.162236 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:24.162462 (+ 226us) client_negotiation.cc:769] Sending connection context
0929 01:02:24.162612 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 01:02:24.162914 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":113,"threads_started":1}
I20260929 01:02:24.164990 3654 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39671
I20260929 01:02:24.165223 3654 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:24.165759 3654 heartbeater.cc:511] Master 127.0.98.62:39671 requested a full tablet report, sending...
I20260929 01:02:24.167273 3463 ts_manager.cc:194] Registered new tserver with Master: 40fd388b4cce46849bb9d16ac46d02eb (127.0.98.1:36407)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 01:02:24.168432 3463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 01:02:24.185870 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39671
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:24.295667 3659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:24.295907 3659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:24.295948 3659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:24.299887 3659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:24.299933 3659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:24.299988 3659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:02:24.305821 3659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:24.306870 3659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:24.307876 3659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:02:24.317287 3659 server_base.cc:1034] running on GCE node
W20260929 01:02:24.317272 3664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:24.317241 3665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:24.317241 3667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:24.317835 3659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:24.318454 3659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:24.319630 3659 hybrid_clock.cc:648] HybridClock initialized: now 1790643744319599 us; error 43 us; skew 500 ppm
Sep 29 01:02:24 dist-test-slave-7zdl krb5kdc[3418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:24.322242 3659 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:02:24.323109 3659 webserver.cc:460] Webserver started at http://127.0.98.2:33063/ using document root <none> and password file <none>
I20260929 01:02:24.323648 3659 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:24.323731 3659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:24.323945 3659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:24.326007 3659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "bc3c7fb2c712459caffb4f6f07c02e93"
format_stamp: "Formatted at 2026-09-29 01:02:24 on dist-test-slave-7zdl"
server_key: "60b8239ea33b290a54e44d7dc8bd71ee"
server_key_iv: "84e8580f17eff155205983627a8b7f92"
server_key_version: "encryptionkey@0"
I20260929 01:02:24.326581 3659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bc3c7fb2c712459caffb4f6f07c02e93"
format_stamp: "Formatted at 2026-09-29 01:02:24 on dist-test-slave-7zdl"
server_key: "60b8239ea33b290a54e44d7dc8bd71ee"
server_key_iv: "84e8580f17eff155205983627a8b7f92"
server_key_version: "encryptionkey@0"
I20260929 01:02:24.330152 3659 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:02:24.332741 3674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:24.333973 3659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:24.334105 3659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "bc3c7fb2c712459caffb4f6f07c02e93"
format_stamp: "Formatted at 2026-09-29 01:02:24 on dist-test-slave-7zdl"
server_key: "60b8239ea33b290a54e44d7dc8bd71ee"
server_key_iv: "84e8580f17eff155205983627a8b7f92"
server_key_version: "encryptionkey@0"
I20260929 01:02:24.334209 3659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:24.369753 3659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:24.373111 3659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:24.373307 3659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:24.374030 3659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:24.374910 3659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:24.374972 3659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:24.375043 3659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:24.375074 3659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:24.383510 3659 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35975
I20260929 01:02:24.383538 3787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35975 every 8 connection(s)
I20260929 01:02:24.384370 3659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:02:24.392174 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 3659
I20260929 01:02:24.392300 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:02:24.392560 392 external_mini_cluster.cc:1428] Setting key 4a9209b4891103207ece6757e2975bc4
Sep 29 01:02:24 dist-test-slave-7zdl krb5kdc[3418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:24.397949 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:24.386107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59485 (local address 127.0.98.62:39671)
0929 01:02:24.386237 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:24.386241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:24.387081 (+ 840us) server_negotiation.cc:407] Connection header received
0929 01:02:24.388486 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:24.388490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:24.388548 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:24.388629 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:24.389762 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:24.390310 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:24.391512 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:24.391742 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:24.391847 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:24.394255 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:24.394258 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:24.394260 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:24.394283 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:24.395780 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:24.396417 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:24.396419 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:24.396435 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:24.396508 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:24.396873 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:24.396875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:24.396876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:24.397036 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:24.397109 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:24.397725 (+ 616us) server_negotiation.cc:299] Negotiation successful
0929 01:02:24.397834 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 01:02:24.399040 3790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:24.386482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39671 (local address 127.0.98.2:59485)
0929 01:02:24.386937 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:24.386978 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:24.388276 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:24.388802 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:24.388809 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:24.389153 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:24.389623 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:24.389636 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:24.390432 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:24.390436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:24.391382 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:24.391391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:24.391879 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:24.391883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:24.392065 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:24.392691 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:24.392707 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:24.394096 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:24.395915 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:24.395922 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:24.395937 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:24.396318 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:24.396592 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:24.396599 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:24.396604 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:24.396786 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:24.397133 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:24.397142 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:24.397453 (+ 311us) client_negotiation.cc:769] Sending connection context
0929 01:02:24.397668 (+ 215us) client_negotiation.cc:240] Negotiation successful
0929 01:02:24.398051 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":105,"threads_started":1}
I20260929 01:02:24.400367 3788 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39671
I20260929 01:02:24.400607 3788 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:24.401064 3788 heartbeater.cc:511] Master 127.0.98.62:39671 requested a full tablet report, sending...
I20260929 01:02:24.402096 3463 ts_manager.cc:194] Registered new tserver with Master: bc3c7fb2c712459caffb4f6f07c02e93 (127.0.98.2:35975)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 01:02:24.402745 3463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 01:02:24.421753 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39671
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:24.529480 3793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:24.529762 3793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:24.529829 3793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:24.533784 3793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:24.533861 3793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:24.533952 3793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:02:24.539088 3793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:24.540089 3793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:24.541081 3793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:24.550472 3801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:24.550449 3799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:24.550494 3798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:24.550668 3793 server_base.cc:1034] running on GCE node
I20260929 01:02:24.551067 3793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:24.551702 3793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:24.552871 3793 hybrid_clock.cc:648] HybridClock initialized: now 1790643744552856 us; error 27 us; skew 500 ppm
Sep 29 01:02:24 dist-test-slave-7zdl krb5kdc[3418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:24.555402 3793 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:02:24.556263 3793 webserver.cc:460] Webserver started at http://127.0.98.3:44909/ using document root <none> and password file <none>
I20260929 01:02:24.556785 3793 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:24.556867 3793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:24.557089 3793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:24.559103 3793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "c0b8c2d758404e2ebb88953d3b4301ad"
format_stamp: "Formatted at 2026-09-29 01:02:24 on dist-test-slave-7zdl"
server_key: "2878e289ac635c621aeaf9e1979e64df"
server_key_iv: "f7a484e71842377b2b859c7750f95127"
server_key_version: "encryptionkey@0"
I20260929 01:02:24.559648 3793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c0b8c2d758404e2ebb88953d3b4301ad"
format_stamp: "Formatted at 2026-09-29 01:02:24 on dist-test-slave-7zdl"
server_key: "2878e289ac635c621aeaf9e1979e64df"
server_key_iv: "f7a484e71842377b2b859c7750f95127"
server_key_version: "encryptionkey@0"
I20260929 01:02:24.563174 3793 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 01:02:24.565395 3808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:24.566646 3793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:24.566763 3793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "c0b8c2d758404e2ebb88953d3b4301ad"
format_stamp: "Formatted at 2026-09-29 01:02:24 on dist-test-slave-7zdl"
server_key: "2878e289ac635c621aeaf9e1979e64df"
server_key_iv: "f7a484e71842377b2b859c7750f95127"
server_key_version: "encryptionkey@0"
I20260929 01:02:24.566905 3793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:24.593565 3793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:24.596410 3793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:24.596608 3793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:24.597187 3793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:24.597977 3793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:24.598052 3793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:24.598121 3793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:24.598161 3793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:24.606832 3793 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33055
I20260929 01:02:24.606849 3921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33055 every 8 connection(s)
I20260929 01:02:24.607791 3793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:02:24.617417 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 3793
I20260929 01:02:24.617574 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
Sep 29 01:02:24 dist-test-slave-7zdl krb5kdc[3418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:24.617903 392 external_mini_cluster.cc:1428] Setting key 0252c8a38649764830c0d3cbbdb44ef5
I20260929 01:02:24.622526 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:24.609530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33955 (local address 127.0.98.62:39671)
0929 01:02:24.609662 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:24.609666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:24.610612 (+ 946us) server_negotiation.cc:407] Connection header received
0929 01:02:24.612071 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:24.612074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:24.612136 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:24.612247 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:24.613453 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:24.614004 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:24.615438 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:24.615665 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:24.615773 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:24.618417 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:24.618421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:24.618424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:24.618451 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:24.620305 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:24.620919 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:24.620922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:24.620925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:24.621042 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:24.621470 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:24.621474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:24.621478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:24.621749 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:24.621852 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:24.622315 (+ 463us) server_negotiation.cc:299] Negotiation successful
0929 01:02:24.622410 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 01:02:24.623378 3924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:24.609902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39671 (local address 127.0.98.3:33955)
0929 01:02:24.610436 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:24.610485 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:24.611847 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:24.612418 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:24.612425 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:24.612803 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:24.613299 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:24.613313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:24.614265 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:24.614270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:24.615312 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:24.615322 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:24.615806 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:24.615810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:24.615975 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:24.616542 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:24.616558 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:24.618256 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:24.620456 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:24.620462 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:24.620476 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:24.620824 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:24.621160 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:24.621163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:24.621165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:24.621355 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:24.621866 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:24.621874 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:24.622113 (+ 239us) client_negotiation.cc:769] Sending connection context
0929 01:02:24.622270 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 01:02:24.622524 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":109,"threads_started":1}
I20260929 01:02:24.624397 3922 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39671
I20260929 01:02:24.624611 3922 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:24.625324 3922 heartbeater.cc:511] Master 127.0.98.62:39671 requested a full tablet report, sending...
I20260929 01:02:24.626466 3463 ts_manager.cc:194] Registered new tserver with Master: c0b8c2d758404e2ebb88953d3b4301ad (127.0.98.3:33055)
I20260929 01:02:24.627076 3463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33955
I20260929 01:02:24.632536 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 01:02:24 dist-test-slave-7zdl krb5kdc[3418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643744, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 01:02:24 dist-test-slave-7zdl krb5kdc[3418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643744, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:24.658708 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:24.650040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.0.98.62:39671)
0929 01:02:24.650203 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:24.650208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:24.650420 (+ 212us) server_negotiation.cc:407] Connection header received
0929 01:02:24.650597 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:24.650600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:24.650641 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:24.650728 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:24.651456 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:24.651998 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:24.652857 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:24.653116 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:24.653230 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:24.655785 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:24.655789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:24.655791 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:24.655836 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:24.657125 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:24.657576 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:24.657579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:24.657580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:24.657662 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:24.657956 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:24.657958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:24.657959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:24.658169 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:24.658270 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:24.658498 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 01:02:24.658592 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 01:02:24.659126 3463 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43952
I20260929 01:02:24.660056 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 3525
I20260929 01:02:24.665328 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 3659
I20260929 01:02:24.670401 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 3793
I20260929 01:02:24.675670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 3431
2026-09-29T01:02:24Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2141 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:02:24 dist-test-slave-7zdl krb5kdc[3934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:02:24 dist-test-slave-7zdl krb5kdc[3934](info): set up 2 sockets
Sep 29 01:02:24 dist-test-slave-7zdl krb5kdc[3934](info): commencing operation
krb5kdc: starting...
W20260929 01:02:25.718719 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:02:25 dist-test-slave-7zdl krb5kdc[3934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:02:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:02:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:02:25.831557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38895
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:25.938966 3947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:25.939203 3947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:25.939244 3947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:25.943152 3947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:02:25.943202 3947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:25.943221 3947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:25.943257 3947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:02:25.943289 3947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:02:25.948434 3947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:25.949399 3947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:25.950366 3947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:25.958442 3955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:25.958540 3953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:25.958477 3952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:25.958607 3947 server_base.cc:1034] running on GCE node
I20260929 01:02:25.959281 3947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:25.960350 3947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:25.961509 3947 hybrid_clock.cc:648] HybridClock initialized: now 1790643745961479 us; error 42 us; skew 500 ppm
Sep 29 01:02:25 dist-test-slave-7zdl krb5kdc[3934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:25.964238 3947 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:02:25.965107 3947 webserver.cc:460] Webserver started at http://127.0.98.62:33325/ using document root <none> and password file <none>
I20260929 01:02:25.965636 3947 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:25.965721 3947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:25.965968 3947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:25.968119 3947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "c485b4cd4e754e70969f0865b08a0873"
format_stamp: "Formatted at 2026-09-29 01:02:25 on dist-test-slave-7zdl"
server_key: "66f6dfbb3ed82379bb1c96dcf28d8869"
server_key_iv: "42be6aaf6961d94abfc5e2222ed73e80"
server_key_version: "encryptionkey@0"
I20260929 01:02:25.968669 3947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "c485b4cd4e754e70969f0865b08a0873"
format_stamp: "Formatted at 2026-09-29 01:02:25 on dist-test-slave-7zdl"
server_key: "66f6dfbb3ed82379bb1c96dcf28d8869"
server_key_iv: "42be6aaf6961d94abfc5e2222ed73e80"
server_key_version: "encryptionkey@0"
I20260929 01:02:25.972149 3947 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:02:25.974320 3962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:25.975533 3947 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:02:25.975660 3947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "c485b4cd4e754e70969f0865b08a0873"
format_stamp: "Formatted at 2026-09-29 01:02:25 on dist-test-slave-7zdl"
server_key: "66f6dfbb3ed82379bb1c96dcf28d8869"
server_key_iv: "42be6aaf6961d94abfc5e2222ed73e80"
server_key_version: "encryptionkey@0"
I20260929 01:02:25.975760 3947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:25.992601 3947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:25.995790 3947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:25.995936 3947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:26.002761 3947 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38895
I20260929 01:02:26.002776 4014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38895 every 8 connection(s)
I20260929 01:02:26.003712 3947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:02:26.006290 4015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:26.007026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 3947
I20260929 01:02:26.007141 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:02:26.007417 392 external_mini_cluster.cc:1428] Setting key 4cdcf59114f209539136bcf6d8a7a243
I20260929 01:02:26.012285 4015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c485b4cd4e754e70969f0865b08a0873: Bootstrap starting.
I20260929 01:02:26.014851 4015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c485b4cd4e754e70969f0865b08a0873: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:26.015744 4015 log.cc:824] T 00000000000000000000000000000000 P c485b4cd4e754e70969f0865b08a0873: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:02:26 dist-test-slave-7zdl krb5kdc[3934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:26.017776 4015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c485b4cd4e754e70969f0865b08a0873: No bootstrap required, opened a new log
I20260929 01:02:26.021265 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:26.008716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50260 (local address 127.0.98.62:38895)
0929 01:02:26.009163 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:26.009178 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:26.009212 (+ 34us) server_negotiation.cc:407] Connection header received
0929 01:02:26.009824 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:26.009839 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:26.010155 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:26.010419 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:26.011322 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:26.012260 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:26.013387 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:26.013826 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:26.013939 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:26.016874 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:26.016881 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:26.016897 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:26.016926 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:26.018627 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:26.019087 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:26.019093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:26.019099 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:26.019211 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:26.019485 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:26.019488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:26.019489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:26.019849 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:26.020056 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:26.020313 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 01:02:26.020585 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":166,"threads_started":1}
I20260929 01:02:26.021138 4015 raft_consensus.cc:348] T 00000000000000000000000000000000 P c485b4cd4e754e70969f0865b08a0873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c485b4cd4e754e70969f0865b08a0873" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38895 } }
I20260929 01:02:26.021376 4015 raft_consensus.cc:374] T 00000000000000000000000000000000 P c485b4cd4e754e70969f0865b08a0873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:26.021436 4015 raft_consensus.cc:729] T 00000000000000000000000000000000 P c485b4cd4e754e70969f0865b08a0873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c485b4cd4e754e70969f0865b08a0873, State: Initialized, Role: FOLLOWER
I20260929 01:02:26.021832 4015 consensus_queue.cc:260] T 00000000000000000000000000000000 P c485b4cd4e754e70969f0865b08a0873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c485b4cd4e754e70969f0865b08a0873" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38895 } }
I20260929 01:02:26.021971 4015 raft_consensus.cc:388] T 00000000000000000000000000000000 P c485b4cd4e754e70969f0865b08a0873 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:02:26.022054 4015 raft_consensus.cc:482] T 00000000000000000000000000000000 P c485b4cd4e754e70969f0865b08a0873 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:02:26.022173 4015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c485b4cd4e754e70969f0865b08a0873 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:26.023049 4015 raft_consensus.cc:504] T 00000000000000000000000000000000 P c485b4cd4e754e70969f0865b08a0873 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c485b4cd4e754e70969f0865b08a0873" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38895 } }
I20260929 01:02:26.023353 4015 leader_election.cc:302] T 00000000000000000000000000000000 P c485b4cd4e754e70969f0865b08a0873 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c485b4cd4e754e70969f0865b08a0873; no voters:
I20260929 01:02:26.023617 4015 leader_election.cc:288] T 00000000000000000000000000000000 P c485b4cd4e754e70969f0865b08a0873 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:02:26.023732 4020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c485b4cd4e754e70969f0865b08a0873 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:26.023958 4020 raft_consensus.cc:686] T 00000000000000000000000000000000 P c485b4cd4e754e70969f0865b08a0873 [term 1 LEADER]: Becoming Leader. State: Replica: c485b4cd4e754e70969f0865b08a0873, State: Running, Role: LEADER
I20260929 01:02:26.024288 4020 consensus_queue.cc:237] T 00000000000000000000000000000000 P c485b4cd4e754e70969f0865b08a0873 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c485b4cd4e754e70969f0865b08a0873" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38895 } }
I20260929 01:02:26.024753 4015 sys_catalog.cc:564] T 00000000000000000000000000000000 P c485b4cd4e754e70969f0865b08a0873 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:02:26.025826 4021 sys_catalog.cc:455] T 00000000000000000000000000000000 P c485b4cd4e754e70969f0865b08a0873 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c485b4cd4e754e70969f0865b08a0873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c485b4cd4e754e70969f0865b08a0873" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38895 } } }
I20260929 01:02:26.025951 4021 sys_catalog.cc:458] T 00000000000000000000000000000000 P c485b4cd4e754e70969f0865b08a0873 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:26.026198 4022 sys_catalog.cc:455] T 00000000000000000000000000000000 P c485b4cd4e754e70969f0865b08a0873 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c485b4cd4e754e70969f0865b08a0873. Latest consensus state: current_term: 1 leader_uuid: "c485b4cd4e754e70969f0865b08a0873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c485b4cd4e754e70969f0865b08a0873" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38895 } } }
I20260929 01:02:26.026283 4029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:02:26.029054 4029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:02:26.026372 4022 sys_catalog.cc:458] T 00000000000000000000000000000000 P c485b4cd4e754e70969f0865b08a0873 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:26.034404 4029 catalog_manager.cc:1347] Generated new cluster ID: c9136e47c93e405887350b6fd2a12629
I20260929 01:02:26.034464 4029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:02:26.047633 4029 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:02:26.048846 4029 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:02:26.064356 4029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c485b4cd4e754e70969f0865b08a0873: Generated new TSK 0
I20260929 01:02:26.065155 4029 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 01:02:26.097074 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38895
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:26.205806 4041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:26.206035 4041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:26.206074 4041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:26.209997 4041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:26.210048 4041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:26.210103 4041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:02:26.215065 4041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:26.215941 4041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:26.216823 4041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:26.225896 4046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:26.225907 4047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:26.225932 4049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:26.226214 4041 server_base.cc:1034] running on GCE node
I20260929 01:02:26.226660 4041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:26.227319 4041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:26.228490 4041 hybrid_clock.cc:648] HybridClock initialized: now 1790643746228483 us; error 35 us; skew 500 ppm
Sep 29 01:02:26 dist-test-slave-7zdl krb5kdc[3934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:26.231060 4041 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:02:26.231817 4041 webserver.cc:460] Webserver started at http://127.0.98.1:34237/ using document root <none> and password file <none>
I20260929 01:02:26.232334 4041 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:26.232414 4041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:26.232632 4041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:26.234491 4041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "1f0f5b6cc22d4f43a2d38ae165bcdf46"
format_stamp: "Formatted at 2026-09-29 01:02:26 on dist-test-slave-7zdl"
server_key: "499fe1fac3c1ae5e7ad3cda75fcdca9e"
server_key_iv: "d55a4c90dfece9edfb8b4c648fa61111"
server_key_version: "encryptionkey@0"
I20260929 01:02:26.235054 4041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1f0f5b6cc22d4f43a2d38ae165bcdf46"
format_stamp: "Formatted at 2026-09-29 01:02:26 on dist-test-slave-7zdl"
server_key: "499fe1fac3c1ae5e7ad3cda75fcdca9e"
server_key_iv: "d55a4c90dfece9edfb8b4c648fa61111"
server_key_version: "encryptionkey@0"
I20260929 01:02:26.238499 4041 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:02:26.240971 4056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:26.242089 4041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:26.242210 4041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "1f0f5b6cc22d4f43a2d38ae165bcdf46"
format_stamp: "Formatted at 2026-09-29 01:02:26 on dist-test-slave-7zdl"
server_key: "499fe1fac3c1ae5e7ad3cda75fcdca9e"
server_key_iv: "d55a4c90dfece9edfb8b4c648fa61111"
server_key_version: "encryptionkey@0"
I20260929 01:02:26.242309 4041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:26.251448 4041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:26.253937 4041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:26.254113 4041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:26.254621 4041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:26.255376 4041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:26.255447 4041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:26.255512 4041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:26.255553 4041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:26.263761 4041 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35203
I20260929 01:02:26.263793 4169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35203 every 8 connection(s)
I20260929 01:02:26.264575 4041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:02:26.272962 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 4041
I20260929 01:02:26.273135 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:02:26.273439 392 external_mini_cluster.cc:1428] Setting key 63b5cbd0e9eb847450f9e78d75e7e0b4
Sep 29 01:02:26 dist-test-slave-7zdl krb5kdc[3934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:26.278342 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:26.266281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44029 (local address 127.0.98.62:38895)
0929 01:02:26.266413 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:26.266417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:26.267214 (+ 797us) server_negotiation.cc:407] Connection header received
0929 01:02:26.268830 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:26.268834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:26.268906 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:26.269016 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:26.270178 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:26.270693 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:26.271808 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:26.272042 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:26.272117 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:26.274685 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:26.274689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:26.274691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:26.274716 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:26.276358 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:26.276827 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:26.276833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:26.276835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:26.276928 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:26.277252 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:26.277257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:26.277258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:26.277452 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:26.277533 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:26.278060 (+ 527us) server_negotiation.cc:299] Negotiation successful
0929 01:02:26.278177 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:02:26.279373 4172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:26.266627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38895 (local address 127.0.98.1:44029)
0929 01:02:26.267067 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:26.267109 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:26.268571 (+ 1462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:26.269230 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:26.269243 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:26.269585 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:26.270053 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:26.270067 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:26.270791 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:26.270836 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:26.271702 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:26.271710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:26.272163 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:26.272166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:26.272296 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:26.272916 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:26.272936 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:26.274506 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:26.276470 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:26.276475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:26.276486 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:26.276730 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:26.277024 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:26.277026 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:26.277028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:26.277163 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:26.277542 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:26.277547 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:26.277820 (+ 273us) client_negotiation.cc:769] Sending connection context
0929 01:02:26.278018 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 01:02:26.278475 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":122,"threads_started":1}
I20260929 01:02:26.280592 4170 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38895
I20260929 01:02:26.280854 4170 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:26.281418 4170 heartbeater.cc:511] Master 127.0.98.62:38895 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 01:02:26.283154 3979 ts_manager.cc:194] Registered new tserver with Master: 1f0f5b6cc22d4f43a2d38ae165bcdf46 (127.0.98.1:35203)
I20260929 01:02:26.284255 3979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44029
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 01:02:26.299535 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38895
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:26.406522 4175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:26.406757 4175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:26.406817 4175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:26.410640 4175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:26.410686 4175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:26.410738 4175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:02:26.415802 4175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:26.416700 4175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:26.417623 4175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:26.426826 4181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:26.426856 4180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:26.426895 4183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:26.427062 4175 server_base.cc:1034] running on GCE node
I20260929 01:02:26.427414 4175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:26.427986 4175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:26.429162 4175 hybrid_clock.cc:648] HybridClock initialized: now 1790643746429153 us; error 32 us; skew 500 ppm
Sep 29 01:02:26 dist-test-slave-7zdl krb5kdc[3934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:26.431739 4175 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:02:26.432549 4175 webserver.cc:460] Webserver started at http://127.0.98.2:36411/ using document root <none> and password file <none>
I20260929 01:02:26.433038 4175 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:26.433094 4175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:26.433246 4175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:26.435170 4175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "264a3c13381643249c1f319a3cb8220f"
format_stamp: "Formatted at 2026-09-29 01:02:26 on dist-test-slave-7zdl"
server_key: "b31e31cb0a06823980f632ba38cffe30"
server_key_iv: "0804e8ebc0863eee8070e10ead332e5e"
server_key_version: "encryptionkey@0"
I20260929 01:02:26.435668 4175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "264a3c13381643249c1f319a3cb8220f"
format_stamp: "Formatted at 2026-09-29 01:02:26 on dist-test-slave-7zdl"
server_key: "b31e31cb0a06823980f632ba38cffe30"
server_key_iv: "0804e8ebc0863eee8070e10ead332e5e"
server_key_version: "encryptionkey@0"
I20260929 01:02:26.439095 4175 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:02:26.441264 4190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:26.442389 4175 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:26.442512 4175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "264a3c13381643249c1f319a3cb8220f"
format_stamp: "Formatted at 2026-09-29 01:02:26 on dist-test-slave-7zdl"
server_key: "b31e31cb0a06823980f632ba38cffe30"
server_key_iv: "0804e8ebc0863eee8070e10ead332e5e"
server_key_version: "encryptionkey@0"
I20260929 01:02:26.442610 4175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:26.456251 4175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:26.459340 4175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:26.459551 4175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:26.460083 4175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:26.460834 4175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:26.460898 4175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:26.460973 4175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:26.461020 4175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:26.469163 4175 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37151
I20260929 01:02:26.469230 4303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37151 every 8 connection(s)
I20260929 01:02:26.470017 4175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:02:26.474895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 4175
I20260929 01:02:26.475031 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:02:26.475335 392 external_mini_cluster.cc:1428] Setting key 99341be1202ca813aadc189012e5d41a
Sep 29 01:02:26 dist-test-slave-7zdl krb5kdc[3934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:26.484483 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:26.471676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40429 (local address 127.0.98.62:38895)
0929 01:02:26.471825 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:26.471831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:26.472627 (+ 796us) server_negotiation.cc:407] Connection header received
0929 01:02:26.474051 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:26.474054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:26.474098 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:26.474169 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:26.475575 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:26.476173 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:26.477442 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:26.477844 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:26.477964 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:26.480913 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:26.480917 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:26.480920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:26.480944 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:26.482301 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:26.482907 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:26.482910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:26.482912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:26.482984 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:26.483370 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:26.483373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:26.483374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:26.483554 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:26.483625 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:26.484257 (+ 632us) server_negotiation.cc:299] Negotiation successful
0929 01:02:26.484353 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 01:02:26.485522 4306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:26.472067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38895 (local address 127.0.98.2:40429)
0929 01:02:26.472479 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:26.472526 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:26.473830 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:26.474321 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:26.474327 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:26.474733 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:26.475426 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:26.475444 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:26.476288 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:26.476291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:26.477315 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:26.477324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:26.477975 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:26.477978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:26.478223 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:26.478904 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:26.478928 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:26.480751 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:26.482412 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:26.482419 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:26.482435 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:26.482789 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:26.483071 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:26.483078 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:26.483080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:26.483269 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:26.483654 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:26.483661 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:26.483972 (+ 311us) client_negotiation.cc:769] Sending connection context
0929 01:02:26.484219 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 01:02:26.484571 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":106,"threads_started":1}
I20260929 01:02:26.486572 4304 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38895
I20260929 01:02:26.486831 4304 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 01:02:26.487303 4304 heartbeater.cc:511] Master 127.0.98.62:38895 requested a full tablet report, sending...
I20260929 01:02:26.488332 3979 ts_manager.cc:194] Registered new tserver with Master: 264a3c13381643249c1f319a3cb8220f (127.0.98.2:37151)
I20260929 01:02:26.488890 3979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 01:02:26.502128 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38895
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:26.610540 4309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:26.610800 4309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:26.610904 4309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:26.614916 4309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:26.614984 4309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:26.615074 4309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:02:26.620203 4309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:26.621196 4309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:26.622175 4309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:26.632436 4315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:26.632489 4309 server_base.cc:1034] running on GCE node
W20260929 01:02:26.632514 4314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:26.632436 4317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:26.633086 4309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:26.633718 4309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:26.634867 4309 hybrid_clock.cc:648] HybridClock initialized: now 1790643746634853 us; error 34 us; skew 500 ppm
Sep 29 01:02:26 dist-test-slave-7zdl krb5kdc[3934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:26.637490 4309 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:02:26.638283 4309 webserver.cc:460] Webserver started at http://127.0.98.3:40609/ using document root <none> and password file <none>
I20260929 01:02:26.638863 4309 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:26.638952 4309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:26.639173 4309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:26.641125 4309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "3ad2a513d28a4e4091863641b4f038eb"
format_stamp: "Formatted at 2026-09-29 01:02:26 on dist-test-slave-7zdl"
server_key: "c1d9650fe13176beb6ae19a844c2907a"
server_key_iv: "48a7aa1add34656bb345fdd9a7a58316"
server_key_version: "encryptionkey@0"
I20260929 01:02:26.641673 4309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3ad2a513d28a4e4091863641b4f038eb"
format_stamp: "Formatted at 2026-09-29 01:02:26 on dist-test-slave-7zdl"
server_key: "c1d9650fe13176beb6ae19a844c2907a"
server_key_iv: "48a7aa1add34656bb345fdd9a7a58316"
server_key_version: "encryptionkey@0"
I20260929 01:02:26.645339 4309 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 01:02:26.648092 4324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:26.649297 4309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:02:26.649437 4309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "3ad2a513d28a4e4091863641b4f038eb"
format_stamp: "Formatted at 2026-09-29 01:02:26 on dist-test-slave-7zdl"
server_key: "c1d9650fe13176beb6ae19a844c2907a"
server_key_iv: "48a7aa1add34656bb345fdd9a7a58316"
server_key_version: "encryptionkey@0"
I20260929 01:02:26.649547 4309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:26.665395 4309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:26.668684 4309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:26.668897 4309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:26.669482 4309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:26.670280 4309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:26.670353 4309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:26.670421 4309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:26.670468 4309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:26.678975 4309 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38559
I20260929 01:02:26.679008 4437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38559 every 8 connection(s)
I20260929 01:02:26.679948 4309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:02:26.687590 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 4309
I20260929 01:02:26.687709 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:02:26.688040 392 external_mini_cluster.cc:1428] Setting key ebf34f25cb1b5c949c8433826ee8ba50
Sep 29 01:02:26 dist-test-slave-7zdl krb5kdc[3934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:26.693267 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:26.681755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33317 (local address 127.0.98.62:38895)
0929 01:02:26.681957 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:26.681962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:26.682775 (+ 813us) server_negotiation.cc:407] Connection header received
0929 01:02:26.684164 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:26.684167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:26.684216 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:26.684293 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:26.685426 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:26.685964 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:26.687107 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:26.687319 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:26.687399 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:26.690133 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:26.690136 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:26.690138 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:26.690161 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:26.691387 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:26.691851 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:26.691856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:26.691858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:26.691938 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:26.692255 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:26.692259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:26.692261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:26.692435 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:26.692531 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:26.692993 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 01:02:26.693106 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260929 01:02:26.693989 4440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:26.682184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38895 (local address 127.0.98.3:33317)
0929 01:02:26.682624 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:26.682664 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:26.683926 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:26.684449 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:26.684455 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:26.684793 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:26.685272 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:26.685285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:26.686069 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:26.686071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:26.687004 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:26.687013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:26.687459 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:26.687463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:26.687668 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:26.688339 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:26.688368 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:26.689974 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:26.691494 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:26.691500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:26.691514 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:26.691754 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:26.692030 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:26.692033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:26.692035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:26.692167 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:26.692540 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:26.692549 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:26.692772 (+ 223us) client_negotiation.cc:769] Sending connection context
0929 01:02:26.692947 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 01:02:26.693242 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":119,"threads_started":1}
I20260929 01:02:26.694975 4438 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38895
I20260929 01:02:26.695211 4438 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:26.695662 4438 heartbeater.cc:511] Master 127.0.98.62:38895 requested a full tablet report, sending...
I20260929 01:02:26.696581 3979 ts_manager.cc:194] Registered new tserver with Master: 3ad2a513d28a4e4091863641b4f038eb (127.0.98.3:38559)
I20260929 01:02:26.697122 3979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33317
I20260929 01:02:26.701665 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 01:02:26 dist-test-slave-7zdl krb5kdc[3934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 01:02:26 dist-test-slave-7zdl krb5kdc[3934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 01:02:26.726245 4445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:26.716971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57300 (local address 127.0.98.1:35203)
0929 01:02:26.717312 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:26.717331 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:26.717353 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:02:26.717402 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:26.717407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:26.717604 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:26.717726 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:26.718334 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:26.719150 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:26.719949 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:26.720307 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:26.720387 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:26.722725 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:26.722734 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:26.722747 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:26.722771 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:26.724435 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:26.724874 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:26.724879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:26.724881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:26.724958 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:26.725257 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:26.725260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:26.725261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:26.725508 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:26.725730 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:26.725911 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 01:02:26.726069 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":147,"threads_started":1}
Sep 29 01:02:26 dist-test-slave-7zdl krb5kdc[3934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:26.736210 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:26.729049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50274 (local address 127.0.98.62:38895)
0929 01:02:26.729186 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:26.729190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:26.729204 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:02:26.729303 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:26.729305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:26.729342 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:26.729407 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:26.729915 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:26.730410 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:26.731192 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:26.731396 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:26.731472 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:26.733534 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:26.733537 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:26.733539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:26.733560 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:26.734777 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:26.735172 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:26.735174 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:26.735176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:26.735243 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:26.735443 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:26.735445 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:26.735446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:26.735637 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:26.735733 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:26.735902 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 01:02:26.736015 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 01:02:26.736577 3979 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50274
I20260929 01:02:26.737142 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 4041
I20260929 01:02:26.742833 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 4175
I20260929 01:02:26.747728 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 4309
I20260929 01:02:26.752931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 3947
2026-09-29T01:02:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2077 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:02:26 dist-test-slave-7zdl krb5kdc[4450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:02:26 dist-test-slave-7zdl krb5kdc[4450](info): set up 2 sockets
Sep 29 01:02:26 dist-test-slave-7zdl krb5kdc[4450](info): commencing operation
krb5kdc: starting...
W20260929 01:02:27.786901 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:02:27 dist-test-slave-7zdl krb5kdc[4450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:02:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:02:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:02:27.903379 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42411
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:28.008775 4463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:28.009040 4463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:28.009109 4463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:28.013099 4463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:02:28.013176 4463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:28.013213 4463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:28.013265 4463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:02:28.013303 4463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:02:28.018290 4463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:28.019306 4463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:28.020251 4463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:28.028108 4469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:28.028173 4471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:28.028216 4468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:28.028275 4463 server_base.cc:1034] running on GCE node
I20260929 01:02:28.028864 4463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:28.029956 4463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:28.031133 4463 hybrid_clock.cc:648] HybridClock initialized: now 1790643748031119 us; error 24 us; skew 500 ppm
Sep 29 01:02:28 dist-test-slave-7zdl krb5kdc[4450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:28.040686 4463 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:02:28.041415 4463 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:28.041518 4463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:28.041747 4463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:28.043853 4463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "f44d2a0f6aa348b1a75f762a96f0b48b"
format_stamp: "Formatted at 2026-09-29 01:02:28 on dist-test-slave-7zdl"
server_key: "78b726b05624e90066a3ab5fad6d8fa1"
server_key_iv: "8bb400cbc040c68c8cff5feda8f47214"
server_key_version: "encryptionkey@0"
I20260929 01:02:28.044417 4463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "f44d2a0f6aa348b1a75f762a96f0b48b"
format_stamp: "Formatted at 2026-09-29 01:02:28 on dist-test-slave-7zdl"
server_key: "78b726b05624e90066a3ab5fad6d8fa1"
server_key_iv: "8bb400cbc040c68c8cff5feda8f47214"
server_key_version: "encryptionkey@0"
I20260929 01:02:28.047952 4463 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:02:28.050302 4477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:28.051554 4463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:02:28.051687 4463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "f44d2a0f6aa348b1a75f762a96f0b48b"
format_stamp: "Formatted at 2026-09-29 01:02:28 on dist-test-slave-7zdl"
server_key: "78b726b05624e90066a3ab5fad6d8fa1"
server_key_iv: "8bb400cbc040c68c8cff5feda8f47214"
server_key_version: "encryptionkey@0"
I20260929 01:02:28.051790 4463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:28.069391 4463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:28.072074 4463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:28.072259 4463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:28.078686 4463 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42411
I20260929 01:02:28.078960 4529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42411 every 8 connection(s)
I20260929 01:02:28.079350 4463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:02:28.081895 4530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:28.087244 4530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f44d2a0f6aa348b1a75f762a96f0b48b: Bootstrap starting.
I20260929 01:02:28.089377 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 4463
I20260929 01:02:28.089519 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:02:28.089591 4530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f44d2a0f6aa348b1a75f762a96f0b48b: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:28.089807 392 external_mini_cluster.cc:1428] Setting key 529d0c9a7c0ec32a4c8981758747a58b
I20260929 01:02:28.090348 4530 log.cc:824] T 00000000000000000000000000000000 P f44d2a0f6aa348b1a75f762a96f0b48b: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:28.092203 4530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f44d2a0f6aa348b1a75f762a96f0b48b: No bootstrap required, opened a new log
I20260929 01:02:28.096232 4530 raft_consensus.cc:348] T 00000000000000000000000000000000 P f44d2a0f6aa348b1a75f762a96f0b48b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44d2a0f6aa348b1a75f762a96f0b48b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42411 } }
I20260929 01:02:28.096510 4530 raft_consensus.cc:374] T 00000000000000000000000000000000 P f44d2a0f6aa348b1a75f762a96f0b48b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:28.096578 4530 raft_consensus.cc:729] T 00000000000000000000000000000000 P f44d2a0f6aa348b1a75f762a96f0b48b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f44d2a0f6aa348b1a75f762a96f0b48b, State: Initialized, Role: FOLLOWER
I20260929 01:02:28.097221 4530 consensus_queue.cc:260] T 00000000000000000000000000000000 P f44d2a0f6aa348b1a75f762a96f0b48b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44d2a0f6aa348b1a75f762a96f0b48b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42411 } }
I20260929 01:02:28.097427 4530 raft_consensus.cc:388] T 00000000000000000000000000000000 P f44d2a0f6aa348b1a75f762a96f0b48b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:02:28.097523 4530 raft_consensus.cc:482] T 00000000000000000000000000000000 P f44d2a0f6aa348b1a75f762a96f0b48b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:02:28.097637 4530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f44d2a0f6aa348b1a75f762a96f0b48b [term 0 FOLLOWER]: Advancing to term 1
Sep 29 01:02:28 dist-test-slave-7zdl krb5kdc[4450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:28.099006 4530 raft_consensus.cc:504] T 00000000000000000000000000000000 P f44d2a0f6aa348b1a75f762a96f0b48b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44d2a0f6aa348b1a75f762a96f0b48b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42411 } }
I20260929 01:02:28.099321 4530 leader_election.cc:302] T 00000000000000000000000000000000 P f44d2a0f6aa348b1a75f762a96f0b48b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f44d2a0f6aa348b1a75f762a96f0b48b; no voters:
I20260929 01:02:28.099828 4530 leader_election.cc:288] T 00000000000000000000000000000000 P f44d2a0f6aa348b1a75f762a96f0b48b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:02:28.099915 4535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f44d2a0f6aa348b1a75f762a96f0b48b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:28.100154 4535 raft_consensus.cc:686] T 00000000000000000000000000000000 P f44d2a0f6aa348b1a75f762a96f0b48b [term 1 LEADER]: Becoming Leader. State: Replica: f44d2a0f6aa348b1a75f762a96f0b48b, State: Running, Role: LEADER
I20260929 01:02:28.100548 4535 consensus_queue.cc:237] T 00000000000000000000000000000000 P f44d2a0f6aa348b1a75f762a96f0b48b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44d2a0f6aa348b1a75f762a96f0b48b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42411 } }
I20260929 01:02:28.101203 4530 sys_catalog.cc:564] T 00000000000000000000000000000000 P f44d2a0f6aa348b1a75f762a96f0b48b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:02:28.102156 4537 sys_catalog.cc:455] T 00000000000000000000000000000000 P f44d2a0f6aa348b1a75f762a96f0b48b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f44d2a0f6aa348b1a75f762a96f0b48b. Latest consensus state: current_term: 1 leader_uuid: "f44d2a0f6aa348b1a75f762a96f0b48b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44d2a0f6aa348b1a75f762a96f0b48b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42411 } } }
I20260929 01:02:28.102352 4537 sys_catalog.cc:458] T 00000000000000000000000000000000 P f44d2a0f6aa348b1a75f762a96f0b48b [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:28.102787 4544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:02:28.103960 4536 sys_catalog.cc:455] T 00000000000000000000000000000000 P f44d2a0f6aa348b1a75f762a96f0b48b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f44d2a0f6aa348b1a75f762a96f0b48b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44d2a0f6aa348b1a75f762a96f0b48b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42411 } } }
I20260929 01:02:28.104154 4536 sys_catalog.cc:458] T 00000000000000000000000000000000 P f44d2a0f6aa348b1a75f762a96f0b48b [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:28.105042 4534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:28.090773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56706 (local address 127.0.98.62:42411)
0929 01:02:28.091167 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:28.091186 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:28.091226 (+ 40us) server_negotiation.cc:407] Connection header received
0929 01:02:28.091809 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:28.091823 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:28.092113 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:28.092435 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:28.093262 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.094193 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.095447 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.095908 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.096021 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:28.098950 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:28.098956 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:28.098976 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:28.099001 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:28.101336 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:28.101876 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:28.101883 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:28.101891 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:28.102051 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:28.102380 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:28.102384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:28.102386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:28.102917 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:28.103104 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:28.103882 (+ 778us) server_negotiation.cc:299] Negotiation successful
0929 01:02:28.104180 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":109,"threads_started":1}
I20260929 01:02:28.106657 4544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:02:28.112104 4544 catalog_manager.cc:1347] Generated new cluster ID: 901164be1dd64feebed380e7f452dd3c
I20260929 01:02:28.112164 4544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:02:28.133085 4544 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:02:28.134443 4544 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:02:28.160300 4544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f44d2a0f6aa348b1a75f762a96f0b48b: Generated new TSK 0
I20260929 01:02:28.161091 4544 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 01:02:28.197968 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42411
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:28.329730 4556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:28.329959 4556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:28.330042 4556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:28.334367 4556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:28.334450 4556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:28.334544 4556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:02:28.339551 4556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:28.340528 4556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:28.341491 4556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:28.350955 4564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:28.350972 4562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:28.350982 4561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:28.351187 4556 server_base.cc:1034] running on GCE node
I20260929 01:02:28.351689 4556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:28.352240 4556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:28.353418 4556 hybrid_clock.cc:648] HybridClock initialized: now 1790643748353411 us; error 31 us; skew 500 ppm
Sep 29 01:02:28 dist-test-slave-7zdl krb5kdc[4450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:28.356352 4556 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:02:28.357021 4556 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:28.357105 4556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:28.357316 4556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:28.359315 4556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "b5dd370dfa2b4c9b993c4c82e65b28e5"
format_stamp: "Formatted at 2026-09-29 01:02:28 on dist-test-slave-7zdl"
server_key: "f6cf345b1f4622de3ffcb61c6bc1617b"
server_key_iv: "f948d3f418909f85e1411042b8b12642"
server_key_version: "encryptionkey@0"
I20260929 01:02:28.359850 4556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b5dd370dfa2b4c9b993c4c82e65b28e5"
format_stamp: "Formatted at 2026-09-29 01:02:28 on dist-test-slave-7zdl"
server_key: "f6cf345b1f4622de3ffcb61c6bc1617b"
server_key_iv: "f948d3f418909f85e1411042b8b12642"
server_key_version: "encryptionkey@0"
I20260929 01:02:28.363317 4556 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:02:28.365556 4570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:28.366677 4556 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:02:28.366840 4556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "b5dd370dfa2b4c9b993c4c82e65b28e5"
format_stamp: "Formatted at 2026-09-29 01:02:28 on dist-test-slave-7zdl"
server_key: "f6cf345b1f4622de3ffcb61c6bc1617b"
server_key_iv: "f948d3f418909f85e1411042b8b12642"
server_key_version: "encryptionkey@0"
I20260929 01:02:28.366946 4556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:28.387993 4556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:28.391229 4556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:28.391414 4556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:28.391956 4556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:28.392743 4556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:28.392812 4556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:28.392876 4556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:28.392923 4556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:28.401544 4556 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46321
I20260929 01:02:28.401602 4683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46321 every 8 connection(s)
I20260929 01:02:28.402179 4556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:02:28.404188 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 4556
I20260929 01:02:28.404318 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:02:28.404600 392 external_mini_cluster.cc:1428] Setting key dce51e71356c08f415d69c3641eb4b51
Sep 29 01:02:28 dist-test-slave-7zdl krb5kdc[4450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 01:02:28.417084 4534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:28.403910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34419 (local address 127.0.98.62:42411)
0929 01:02:28.404069 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:28.404073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:28.404956 (+ 883us) server_negotiation.cc:407] Connection header received
0929 01:02:28.406485 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:28.406489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:28.406539 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:28.406616 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:28.408082 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.408594 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.410342 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.410561 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.410641 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:28.413372 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:28.413376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:28.413378 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:28.413401 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:28.414863 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:28.415444 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:28.415448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:28.415450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:28.415527 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:28.415860 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:28.415863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:28.415865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:28.416105 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:28.416204 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:28.416792 (+ 588us) server_negotiation.cc:299] Negotiation successful
0929 01:02:28.416893 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 01:02:28.417902 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:28.404341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42411 (local address 127.0.98.1:34419)
0929 01:02:28.404813 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:28.404863 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:28.406279 (+ 1416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:28.406783 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:28.406790 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:28.407179 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:28.407890 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:28.407910 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.408702 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.408708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:28.410228 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:28.410240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.410711 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.410716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:28.410991 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:28.411594 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:28.411621 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:28.413212 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:28.414969 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:28.414974 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:28.414989 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:28.415344 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:28.415621 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:28.415624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:28.415626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:28.415760 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:28.416213 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:28.416222 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:28.416514 (+ 292us) client_negotiation.cc:769] Sending connection context
0929 01:02:28.416742 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 01:02:28.417069 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":134,"threads_started":1}
I20260929 01:02:28.419226 4684 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42411
I20260929 01:02:28.419452 4684 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:28.419875 4684 heartbeater.cc:511] Master 127.0.98.62:42411 requested a full tablet report, sending...
I20260929 01:02:28.421288 4494 ts_manager.cc:194] Registered new tserver with Master: b5dd370dfa2b4c9b993c4c82e65b28e5 (127.0.98.1:46321)
I20260929 01:02:28.422363 4494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 01:02:28.431993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42411
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:28.537639 4689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:28.537875 4689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:28.537926 4689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:28.541766 4689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:28.541810 4689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:28.541867 4689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:02:28.547184 4689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:28.548128 4689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:28.549074 4689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:28.561105 4695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:28.561297 4694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:28.561345 4697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:28.562440 4689 server_base.cc:1034] running on GCE node
I20260929 01:02:28.563143 4689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:28.563791 4689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:28.564966 4689 hybrid_clock.cc:648] HybridClock initialized: now 1790643748564957 us; error 27 us; skew 500 ppm
Sep 29 01:02:28 dist-test-slave-7zdl krb5kdc[4450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:28.568254 4689 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:02:28.569028 4689 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:28.569116 4689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:28.569365 4689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:28.572360 4689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "12c2a71b7f1c4f84989af10986bdfcde"
format_stamp: "Formatted at 2026-09-29 01:02:28 on dist-test-slave-7zdl"
server_key: "ad7129fdc7b5742abaf4e31b802f3087"
server_key_iv: "416d60ac6eb03559208c7d5a8cb98b45"
server_key_version: "encryptionkey@0"
I20260929 01:02:28.573066 4689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "12c2a71b7f1c4f84989af10986bdfcde"
format_stamp: "Formatted at 2026-09-29 01:02:28 on dist-test-slave-7zdl"
server_key: "ad7129fdc7b5742abaf4e31b802f3087"
server_key_iv: "416d60ac6eb03559208c7d5a8cb98b45"
server_key_version: "encryptionkey@0"
I20260929 01:02:28.577829 4689 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 01:02:28.581175 4703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:28.582892 4689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:02:28.583015 4689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "12c2a71b7f1c4f84989af10986bdfcde"
format_stamp: "Formatted at 2026-09-29 01:02:28 on dist-test-slave-7zdl"
server_key: "ad7129fdc7b5742abaf4e31b802f3087"
server_key_iv: "416d60ac6eb03559208c7d5a8cb98b45"
server_key_version: "encryptionkey@0"
I20260929 01:02:28.583129 4689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:28.603821 4689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:28.607127 4689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:28.607334 4689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:28.607901 4689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:28.608709 4689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:28.608774 4689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:28.608841 4689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:28.608886 4689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:28.617184 4689 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44215
I20260929 01:02:28.617201 4816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44215 every 8 connection(s)
I20260929 01:02:28.617764 4689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 01:02:28 dist-test-slave-7zdl krb5kdc[4450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:28.627643 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 4689
I20260929 01:02:28.627758 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:02:28.628034 392 external_mini_cluster.cc:1428] Setting key 875b03d7ed9f5e0090dec931aa051aad
I20260929 01:02:28.631196 4534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:28.619453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54759 (local address 127.0.98.62:42411)
0929 01:02:28.619582 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:28.619586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:28.620413 (+ 827us) server_negotiation.cc:407] Connection header received
0929 01:02:28.621834 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:28.621837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:28.621895 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:28.622004 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:28.623207 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.623749 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.624897 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.625123 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.625231 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:28.627565 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:28.627568 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:28.627571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:28.627594 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:28.629206 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:28.629694 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:28.629698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:28.629700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:28.629782 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:28.630105 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:28.630110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:28.630112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:28.630289 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:28.630392 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:28.630901 (+ 509us) server_negotiation.cc:299] Negotiation successful
0929 01:02:28.631028 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 01:02:28.631985 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:28.619825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42411 (local address 127.0.98.2:54759)
0929 01:02:28.620268 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:28.620308 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:28.621618 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:28.622167 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:28.622174 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:28.622528 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:28.623071 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:28.623085 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.623867 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.623871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:28.624749 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:28.624758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.625266 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.625270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:28.625431 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:28.625948 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:28.625964 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:28.627384 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:28.629318 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:28.629324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:28.629338 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:28.629594 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:28.629873 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:28.629876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:28.629878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:28.630014 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:28.630388 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:28.630394 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:28.630641 (+ 247us) client_negotiation.cc:769] Sending connection context
0929 01:02:28.630847 (+ 206us) client_negotiation.cc:240] Negotiation successful
0929 01:02:28.631165 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":108,"threads_started":1}
I20260929 01:02:28.633097 4817 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42411
I20260929 01:02:28.633355 4817 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:28.633888 4817 heartbeater.cc:511] Master 127.0.98.62:42411 requested a full tablet report, sending...
I20260929 01:02:28.635092 4494 ts_manager.cc:194] Registered new tserver with Master: 12c2a71b7f1c4f84989af10986bdfcde (127.0.98.2:44215)
I20260929 01:02:28.635888 4494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54759
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 01:02:28.657455 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42411
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:28.765241 4822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:28.765470 4822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:28.765512 4822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:28.769424 4822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:28.769476 4822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:28.769534 4822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:02:28.774863 4822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:28.775784 4822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:28.776698 4822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:28.785996 4828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:28.786010 4827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:28.786031 4830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:28.786109 4822 server_base.cc:1034] running on GCE node
I20260929 01:02:28.786705 4822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:28.787412 4822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:28.788565 4822 hybrid_clock.cc:648] HybridClock initialized: now 1790643748788548 us; error 29 us; skew 500 ppm
Sep 29 01:02:28 dist-test-slave-7zdl krb5kdc[4450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:28.791205 4822 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:02:28.791894 4822 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:28.791953 4822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:28.792201 4822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:28.794175 4822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "7173b2474809425497f7384daf982d2d"
format_stamp: "Formatted at 2026-09-29 01:02:28 on dist-test-slave-7zdl"
server_key: "0fbe51aa11e4d19697cc2d7387e0d25d"
server_key_iv: "f9d3fd6fb45fcc5368389c9e16086184"
server_key_version: "encryptionkey@0"
I20260929 01:02:28.794737 4822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7173b2474809425497f7384daf982d2d"
format_stamp: "Formatted at 2026-09-29 01:02:28 on dist-test-slave-7zdl"
server_key: "0fbe51aa11e4d19697cc2d7387e0d25d"
server_key_iv: "f9d3fd6fb45fcc5368389c9e16086184"
server_key_version: "encryptionkey@0"
I20260929 01:02:28.798188 4822 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:02:28.800488 4836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:28.801746 4822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260929 01:02:28.801862 4822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "7173b2474809425497f7384daf982d2d"
format_stamp: "Formatted at 2026-09-29 01:02:28 on dist-test-slave-7zdl"
server_key: "0fbe51aa11e4d19697cc2d7387e0d25d"
server_key_iv: "f9d3fd6fb45fcc5368389c9e16086184"
server_key_version: "encryptionkey@0"
I20260929 01:02:28.801954 4822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:28.820676 4822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:28.823400 4822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:28.823582 4822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:28.824169 4822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:28.825006 4822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:28.825052 4822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:28.825089 4822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:28.825138 4822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:28.833514 4822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34091
I20260929 01:02:28.833563 4949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34091 every 8 connection(s)
I20260929 01:02:28.834100 4822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
Sep 29 01:02:28 dist-test-slave-7zdl krb5kdc[4450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:28.843555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 4822
I20260929 01:02:28.843694 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:02:28.844022 392 external_mini_cluster.cc:1428] Setting key 25947b803bcefbbcbde60759adcaf877
I20260929 01:02:28.847401 4534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:28.835742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42571 (local address 127.0.98.62:42411)
0929 01:02:28.835880 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:28.835884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:28.836670 (+ 786us) server_negotiation.cc:407] Connection header received
0929 01:02:28.837986 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:28.837989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:28.838049 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:28.838151 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:28.839273 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.839805 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.840921 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.841138 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.841245 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:28.843626 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:28.843629 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:28.843631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:28.843671 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:28.845442 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:28.846022 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:28.846026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:28.846027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:28.846142 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:28.846437 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:28.846440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:28.846441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:28.846619 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:28.846693 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:28.847186 (+ 493us) server_negotiation.cc:299] Negotiation successful
0929 01:02:28.847286 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:02:28.848217 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:28.836106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42411 (local address 127.0.98.3:42571)
0929 01:02:28.836536 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:28.836575 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:28.837784 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:28.838312 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:28.838319 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:28.838663 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:28.839131 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:28.839144 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.839924 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.839931 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:28.840788 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:28.840797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.841281 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.841284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:28.841455 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:28.841999 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:28.842014 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:28.843465 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:28.845549 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:28.845555 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:28.845569 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:28.845895 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:28.846230 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:28.846232 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:28.846234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:28.846360 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:28.846699 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:28.846703 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:28.846987 (+ 284us) client_negotiation.cc:769] Sending connection context
0929 01:02:28.847141 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 01:02:28.847391 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":103,"threads_started":1}
I20260929 01:02:28.849186 4950 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42411
I20260929 01:02:28.849406 4950 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:28.849825 4950 heartbeater.cc:511] Master 127.0.98.62:42411 requested a full tablet report, sending...
I20260929 01:02:28.850706 4494 ts_manager.cc:194] Registered new tserver with Master: 7173b2474809425497f7384daf982d2d (127.0.98.3:34091)
I20260929 01:02:28.851259 4494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42571
I20260929 01:02:28.857501 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:02:28.865708 4534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:28.859318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56720 (local address 127.0.98.62:42411)
0929 01:02:28.859425 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:28.859429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:28.859570 (+ 141us) server_negotiation.cc:407] Connection header received
0929 01:02:28.859655 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:28.859657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:28.859690 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:28.859752 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:28.860379 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.860861 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.861620 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.861813 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.861885 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:28.863075 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:28.863078 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:28.863080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:28.863099 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:28.864261 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:28.864613 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:28.864616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:28.864617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:28.864701 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:28.864908 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:28.864910 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:28.864911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:28.865099 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:28.865236 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:28.865458 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 01:02:28.865559 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 01:02:28.868331 4494 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:02:28.871425 4494 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:02:28.885166 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:28.880001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53060 (local address 127.0.98.1:46321)
0929 01:02:28.880697 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:28.880704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:28.880724 (+ 20us) server_negotiation.cc:407] Connection header received
0929 01:02:28.880792 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:28.880796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:28.880989 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:28.881139 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:28.881348 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.882165 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.883799 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.884440 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.884549 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:28.884689 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:28.884889 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 01:02:28.884991 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":101,"threads_started":1}
I20260929 01:02:28.885152 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:28.879387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46321 (local address 127.0.0.1:53060)
0929 01:02:28.879808 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:28.879821 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:28.879900 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:28.881125 (+ 1225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:28.881128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:28.881145 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:28.881238 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:28.881245 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.882342 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.882346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:28.883689 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:28.883698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.884568 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.884572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:28.884690 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:28.884743 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 01:02:28.884851 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 01:02:28.884981 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":31,"thread_start_us":99,"threads_started":1}
I20260929 01:02:28.885152 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:28.879072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34091 (local address 127.0.0.1:42118)
0929 01:02:28.879422 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:28.879472 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:28.879593 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:28.880714 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:28.880720 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:28.880743 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:28.880878 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:28.880884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.882276 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.882282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:28.883718 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:28.883728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.884754 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.884770 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:28.884878 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:28.884898 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 01:02:28.884941 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 01:02:28.885008 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":40,"thread_start_us":78,"threads_started":1}
I20260929 01:02:28.885746 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:28.879810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44215 (local address 127.0.0.1:37790)
0929 01:02:28.880390 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:28.880401 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:28.880470 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:28.881414 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:28.881416 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:28.881434 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:28.881501 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:28.881507 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.882900 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.882904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:28.884166 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:28.884175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.885351 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.885355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:28.885462 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:28.885490 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 01:02:28.885560 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 01:02:28.885628 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":90,"threads_started":1}
I20260929 01:02:28.885773 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:28.879201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42118 (local address 127.0.98.3:34091)
0929 01:02:28.880258 (+ 1057us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:28.880272 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:28.880294 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:02:28.880366 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:28.880372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:28.880573 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:28.880725 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:28.880998 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.882142 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.883870 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.884627 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.885226 (+ 599us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:28.885371 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:28.885433 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 01:02:28.885645 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":965,"thread_start_us":205,"threads_started":1}
I20260929 01:02:28.886242 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:28.879978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37790 (local address 127.0.98.2:44215)
0929 01:02:28.881023 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:28.881036 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:28.881057 (+ 21us) server_negotiation.cc:407] Connection header received
0929 01:02:28.881126 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:28.881130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:28.881291 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:28.881439 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:28.881578 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.882646 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.884285 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:28.885044 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:28.885142 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:28.885290 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:28.885993 (+ 703us) server_negotiation.cc:299] Negotiation successful
0929 01:02:28.886087 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":935,"thread_start_us":137,"threads_started":1}
I20260929 01:02:28.888490 4884 tablet_service.cc:1467] Processing CreateTablet for tablet 5d6a3fcfca8047dc94314c42adb241ac (DEFAULT_TABLE table=test-table [id=7141766042bb42bfbaacf8891b64a052]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:28.888489 4618 tablet_service.cc:1467] Processing CreateTablet for tablet 5d6a3fcfca8047dc94314c42adb241ac (DEFAULT_TABLE table=test-table [id=7141766042bb42bfbaacf8891b64a052]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:28.888666 4751 tablet_service.cc:1467] Processing CreateTablet for tablet 5d6a3fcfca8047dc94314c42adb241ac (DEFAULT_TABLE table=test-table [id=7141766042bb42bfbaacf8891b64a052]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:28.889521 4751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d6a3fcfca8047dc94314c42adb241ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:28.889549 4618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d6a3fcfca8047dc94314c42adb241ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:28.889554 4884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d6a3fcfca8047dc94314c42adb241ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:28.895954 4967 tablet_bootstrap.cc:492] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde: Bootstrap starting.
I20260929 01:02:28.897013 4969 tablet_bootstrap.cc:492] T 5d6a3fcfca8047dc94314c42adb241ac P 7173b2474809425497f7384daf982d2d: Bootstrap starting.
I20260929 01:02:28.897996 4967 tablet_bootstrap.cc:654] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:28.898653 4969 tablet_bootstrap.cc:654] T 5d6a3fcfca8047dc94314c42adb241ac P 7173b2474809425497f7384daf982d2d: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:28.899091 4967 log.cc:824] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:28.899166 4968 tablet_bootstrap.cc:492] T 5d6a3fcfca8047dc94314c42adb241ac P b5dd370dfa2b4c9b993c4c82e65b28e5: Bootstrap starting.
I20260929 01:02:28.900002 4969 log.cc:824] T 5d6a3fcfca8047dc94314c42adb241ac P 7173b2474809425497f7384daf982d2d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:28.900987 4967 tablet_bootstrap.cc:492] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde: No bootstrap required, opened a new log
I20260929 01:02:28.901173 4967 ts_tablet_manager.cc:1397] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 01:02:28.901322 4968 tablet_bootstrap.cc:654] T 5d6a3fcfca8047dc94314c42adb241ac P b5dd370dfa2b4c9b993c4c82e65b28e5: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:28.902506 4969 tablet_bootstrap.cc:492] T 5d6a3fcfca8047dc94314c42adb241ac P 7173b2474809425497f7384daf982d2d: No bootstrap required, opened a new log
I20260929 01:02:28.902630 4968 log.cc:824] T 5d6a3fcfca8047dc94314c42adb241ac P b5dd370dfa2b4c9b993c4c82e65b28e5: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:28.902691 4969 ts_tablet_manager.cc:1397] T 5d6a3fcfca8047dc94314c42adb241ac P 7173b2474809425497f7384daf982d2d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 01:02:28.904583 4968 tablet_bootstrap.cc:492] T 5d6a3fcfca8047dc94314c42adb241ac P b5dd370dfa2b4c9b993c4c82e65b28e5: No bootstrap required, opened a new log
I20260929 01:02:28.904762 4968 ts_tablet_manager.cc:1397] T 5d6a3fcfca8047dc94314c42adb241ac P b5dd370dfa2b4c9b993c4c82e65b28e5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 01:02:28.904608 4967 raft_consensus.cc:348] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7173b2474809425497f7384daf982d2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34091 } } peers { permanent_uuid: "12c2a71b7f1c4f84989af10986bdfcde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44215 } } peers { permanent_uuid: "b5dd370dfa2b4c9b993c4c82e65b28e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46321 } }
I20260929 01:02:28.904893 4967 raft_consensus.cc:374] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:28.904958 4967 raft_consensus.cc:729] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c2a71b7f1c4f84989af10986bdfcde, State: Initialized, Role: FOLLOWER
I20260929 01:02:28.905434 4967 consensus_queue.cc:260] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7173b2474809425497f7384daf982d2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34091 } } peers { permanent_uuid: "12c2a71b7f1c4f84989af10986bdfcde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44215 } } peers { permanent_uuid: "b5dd370dfa2b4c9b993c4c82e65b28e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46321 } }
I20260929 01:02:28.906111 4817 heartbeater.cc:503] Master 127.0.98.62:42411 was elected leader, sending a full tablet report...
I20260929 01:02:28.906373 4967 ts_tablet_manager.cc:1428] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260929 01:02:28.906680 4969 raft_consensus.cc:348] T 5d6a3fcfca8047dc94314c42adb241ac P 7173b2474809425497f7384daf982d2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7173b2474809425497f7384daf982d2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34091 } } peers { permanent_uuid: "12c2a71b7f1c4f84989af10986bdfcde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44215 } } peers { permanent_uuid: "b5dd370dfa2b4c9b993c4c82e65b28e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46321 } }
I20260929 01:02:28.906977 4969 raft_consensus.cc:374] T 5d6a3fcfca8047dc94314c42adb241ac P 7173b2474809425497f7384daf982d2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:28.907032 4969 raft_consensus.cc:729] T 5d6a3fcfca8047dc94314c42adb241ac P 7173b2474809425497f7384daf982d2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7173b2474809425497f7384daf982d2d, State: Initialized, Role: FOLLOWER
I20260929 01:02:28.907863 4969 consensus_queue.cc:260] T 5d6a3fcfca8047dc94314c42adb241ac P 7173b2474809425497f7384daf982d2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7173b2474809425497f7384daf982d2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34091 } } peers { permanent_uuid: "12c2a71b7f1c4f84989af10986bdfcde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44215 } } peers { permanent_uuid: "b5dd370dfa2b4c9b993c4c82e65b28e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46321 } }
I20260929 01:02:28.908536 4950 heartbeater.cc:503] Master 127.0.98.62:42411 was elected leader, sending a full tablet report...
I20260929 01:02:28.908538 4968 raft_consensus.cc:348] T 5d6a3fcfca8047dc94314c42adb241ac P b5dd370dfa2b4c9b993c4c82e65b28e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7173b2474809425497f7384daf982d2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34091 } } peers { permanent_uuid: "12c2a71b7f1c4f84989af10986bdfcde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44215 } } peers { permanent_uuid: "b5dd370dfa2b4c9b993c4c82e65b28e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46321 } }
I20260929 01:02:28.908754 4968 raft_consensus.cc:374] T 5d6a3fcfca8047dc94314c42adb241ac P b5dd370dfa2b4c9b993c4c82e65b28e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:28.908766 4969 ts_tablet_manager.cc:1428] T 5d6a3fcfca8047dc94314c42adb241ac P 7173b2474809425497f7384daf982d2d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 01:02:28.908813 4968 raft_consensus.cc:729] T 5d6a3fcfca8047dc94314c42adb241ac P b5dd370dfa2b4c9b993c4c82e65b28e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5dd370dfa2b4c9b993c4c82e65b28e5, State: Initialized, Role: FOLLOWER
I20260929 01:02:28.909219 4968 consensus_queue.cc:260] T 5d6a3fcfca8047dc94314c42adb241ac P b5dd370dfa2b4c9b993c4c82e65b28e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7173b2474809425497f7384daf982d2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34091 } } peers { permanent_uuid: "12c2a71b7f1c4f84989af10986bdfcde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44215 } } peers { permanent_uuid: "b5dd370dfa2b4c9b993c4c82e65b28e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46321 } }
I20260929 01:02:28.909816 4684 heartbeater.cc:503] Master 127.0.98.62:42411 was elected leader, sending a full tablet report...
I20260929 01:02:28.911001 4968 ts_tablet_manager.cc:1428] T 5d6a3fcfca8047dc94314c42adb241ac P b5dd370dfa2b4c9b993c4c82e65b28e5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 01:02:29.031889 4973 raft_consensus.cc:482] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:02:29.032076 4973 raft_consensus.cc:504] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7173b2474809425497f7384daf982d2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34091 } } peers { permanent_uuid: "12c2a71b7f1c4f84989af10986bdfcde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44215 } } peers { permanent_uuid: "b5dd370dfa2b4c9b993c4c82e65b28e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46321 } }
I20260929 01:02:29.033154 4973 leader_election.cc:288] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7173b2474809425497f7384daf982d2d (127.0.98.3:34091), b5dd370dfa2b4c9b993c4c82e65b28e5 (127.0.98.1:46321)
I20260929 01:02:29.036947 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.033367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46321 (local address 127.0.98.2:51015)
0929 01:02:29.033505 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:29.033523 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:29.033623 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:29.033877 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:29.033879 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:29.033895 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:29.034043 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:29.034051 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:29.034908 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:29.034911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:29.035841 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:29.035849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:29.036553 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:29.036557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:29.036660 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:29.036688 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 01:02:29.036761 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 01:02:29.036826 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":22}
I20260929 01:02:29.037003 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.033501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51015 (local address 127.0.98.1:46321)
0929 01:02:29.033653 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:29.033660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:29.033677 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:02:29.033739 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:29.033741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:29.033780 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:29.033864 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:29.034143 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:29.034760 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:29.035967 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:29.036443 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:29.036520 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:29.036568 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:29.036784 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 01:02:29.036849 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 01:02:29.037199 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.033367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34091 (local address 127.0.98.2:47401)
0929 01:02:29.033916 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:29.033927 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:29.034013 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:29.034251 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:29.034254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:29.034270 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:29.034382 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:29.034388 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:29.035198 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:29.035201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:29.036091 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:29.036100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:29.036794 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:29.036797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:29.036887 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:29.036908 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 01:02:29.036981 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 01:02:29.037049 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":235,"thread_start_us":139,"threads_started":1}
I20260929 01:02:29.037237 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.033501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47401 (local address 127.0.98.3:34091)
0929 01:02:29.033649 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:29.033656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:29.034013 (+ 357us) server_negotiation.cc:407] Connection header received
0929 01:02:29.034090 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:29.034095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:29.034157 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:29.034231 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:29.034497 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:29.035094 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:29.036201 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:29.036653 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:29.036771 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:29.036823 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:29.037019 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 01:02:29.037093 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 01:02:29.037644 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6a3fcfca8047dc94314c42adb241ac" candidate_uuid: "12c2a71b7f1c4f84989af10986bdfcde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5dd370dfa2b4c9b993c4c82e65b28e5" is_pre_election: true
I20260929 01:02:29.037789 4904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6a3fcfca8047dc94314c42adb241ac" candidate_uuid: "12c2a71b7f1c4f84989af10986bdfcde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7173b2474809425497f7384daf982d2d" is_pre_election: true
I20260929 01:02:29.037950 4638 raft_consensus.cc:2454] T 5d6a3fcfca8047dc94314c42adb241ac P b5dd370dfa2b4c9b993c4c82e65b28e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12c2a71b7f1c4f84989af10986bdfcde in term 0.
I20260929 01:02:29.038087 4904 raft_consensus.cc:2454] T 5d6a3fcfca8047dc94314c42adb241ac P 7173b2474809425497f7384daf982d2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12c2a71b7f1c4f84989af10986bdfcde in term 0.
I20260929 01:02:29.038450 4705 leader_election.cc:302] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c2a71b7f1c4f84989af10986bdfcde, b5dd370dfa2b4c9b993c4c82e65b28e5; no voters:
I20260929 01:02:29.038717 4973 raft_consensus.cc:2781] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:02:29.038825 4973 raft_consensus.cc:482] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:02:29.038887 4973 raft_consensus.cc:3037] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:29.039897 4973 raft_consensus.cc:504] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7173b2474809425497f7384daf982d2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34091 } } peers { permanent_uuid: "12c2a71b7f1c4f84989af10986bdfcde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44215 } } peers { permanent_uuid: "b5dd370dfa2b4c9b993c4c82e65b28e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46321 } }
I20260929 01:02:29.040258 4973 leader_election.cc:288] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde [CANDIDATE]: Term 1 election: Requested vote from peers 7173b2474809425497f7384daf982d2d (127.0.98.3:34091), b5dd370dfa2b4c9b993c4c82e65b28e5 (127.0.98.1:46321)
I20260929 01:02:29.040540 4904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6a3fcfca8047dc94314c42adb241ac" candidate_uuid: "12c2a71b7f1c4f84989af10986bdfcde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7173b2474809425497f7384daf982d2d"
I20260929 01:02:29.040654 4638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6a3fcfca8047dc94314c42adb241ac" candidate_uuid: "12c2a71b7f1c4f84989af10986bdfcde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5dd370dfa2b4c9b993c4c82e65b28e5"
I20260929 01:02:29.040643 4904 raft_consensus.cc:3037] T 5d6a3fcfca8047dc94314c42adb241ac P 7173b2474809425497f7384daf982d2d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:29.040782 4638 raft_consensus.cc:3037] T 5d6a3fcfca8047dc94314c42adb241ac P b5dd370dfa2b4c9b993c4c82e65b28e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:29.041688 4904 raft_consensus.cc:2454] T 5d6a3fcfca8047dc94314c42adb241ac P 7173b2474809425497f7384daf982d2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12c2a71b7f1c4f84989af10986bdfcde in term 1.
I20260929 01:02:29.041849 4638 raft_consensus.cc:2454] T 5d6a3fcfca8047dc94314c42adb241ac P b5dd370dfa2b4c9b993c4c82e65b28e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12c2a71b7f1c4f84989af10986bdfcde in term 1.
I20260929 01:02:29.041989 4704 leader_election.cc:302] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c2a71b7f1c4f84989af10986bdfcde, 7173b2474809425497f7384daf982d2d; no voters:
I20260929 01:02:29.042148 4973 raft_consensus.cc:2781] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:29.042368 4973 raft_consensus.cc:686] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde [term 1 LEADER]: Becoming Leader. State: Replica: 12c2a71b7f1c4f84989af10986bdfcde, State: Running, Role: LEADER
I20260929 01:02:29.042693 4973 consensus_queue.cc:237] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7173b2474809425497f7384daf982d2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34091 } } peers { permanent_uuid: "12c2a71b7f1c4f84989af10986bdfcde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44215 } } peers { permanent_uuid: "b5dd370dfa2b4c9b993c4c82e65b28e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46321 } }
I20260929 01:02:29.045644 4494 catalog_manager.cc:5495] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde reported cstate change: term changed from 0 to 1, leader changed from <none> to 12c2a71b7f1c4f84989af10986bdfcde (127.0.98.2). New cstate: current_term: 1 leader_uuid: "12c2a71b7f1c4f84989af10986bdfcde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7173b2474809425497f7384daf982d2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12c2a71b7f1c4f84989af10986bdfcde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5dd370dfa2b4c9b993c4c82e65b28e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46321 } health_report { overall_health: UNKNOWN } } }
I20260929 01:02:29.070951 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.067605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.0.98.2:44215)
0929 01:02:29.067721 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:29.067725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:29.067738 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:02:29.067875 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:29.067878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:29.067928 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:29.068019 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:02:29.068222 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:29.068708 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:29.069535 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:29.069730 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:29.069801 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:29.070063 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:02:29.070525 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:02:29.070614 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:29.070698 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 01:02:29.070741 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 01:02:29.076361 4638 raft_consensus.cc:1261] T 5d6a3fcfca8047dc94314c42adb241ac P b5dd370dfa2b4c9b993c4c82e65b28e5 [term 1 FOLLOWER]: Refusing update from remote peer 12c2a71b7f1c4f84989af10986bdfcde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:29.076361 4904 raft_consensus.cc:1261] T 5d6a3fcfca8047dc94314c42adb241ac P 7173b2474809425497f7384daf982d2d [term 1 FOLLOWER]: Refusing update from remote peer 12c2a71b7f1c4f84989af10986bdfcde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:29.077020 4973 consensus_queue.cc:1035] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde [LEADER]: Connected to new peer: Peer: permanent_uuid: "7173b2474809425497f7384daf982d2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:29.077198 4977 consensus_queue.cc:1035] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5dd370dfa2b4c9b993c4c82e65b28e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:29.084482 4980 mvcc.cc:205] Tried to move back new op lower bound from 7334476796210380800 to 7334476796081262592. Current Snapshot: MvccSnapshot[applied={T|T < 7334476796210380800}]
I20260929 01:02:29.084843 4981 mvcc.cc:205] Tried to move back new op lower bound from 7334476796210380800 to 7334476796081262592. Current Snapshot: MvccSnapshot[applied={T|T < 7334476796210380800}]
I20260929 01:02:29.093750 4493 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:29.093924 4493 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:29.094561 4982 mvcc.cc:205] Tried to move back new op lower bound from 7334476796210380800 to 7334476796081262592. Current Snapshot: MvccSnapshot[applied={T|T < 7334476796210380800}]
I20260929 01:02:29.096377 4493 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f44d2a0f6aa348b1a75f762a96f0b48b: Sending DeleteTablet for 3 replicas of tablet 5d6a3fcfca8047dc94314c42adb241ac
I20260929 01:02:29.097060 4884 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d6a3fcfca8047dc94314c42adb241ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42118
I20260929 01:02:29.097389 4751 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d6a3fcfca8047dc94314c42adb241ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37790
I20260929 01:02:29.097594 4618 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d6a3fcfca8047dc94314c42adb241ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53060
I20260929 01:02:29.097874 4990 tablet_replica.cc:330] T 5d6a3fcfca8047dc94314c42adb241ac P 7173b2474809425497f7384daf982d2d: stopping tablet replica
I20260929 01:02:29.098167 4990 raft_consensus.cc:2229] T 5d6a3fcfca8047dc94314c42adb241ac P 7173b2474809425497f7384daf982d2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:29.098474 4990 raft_consensus.cc:2258] T 5d6a3fcfca8047dc94314c42adb241ac P 7173b2474809425497f7384daf982d2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:29.098841 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 4556
I20260929 01:02:29.100041 4990 ts_tablet_manager.cc:1905] T 5d6a3fcfca8047dc94314c42adb241ac P 7173b2474809425497f7384daf982d2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:29.102918 4990 ts_tablet_manager.cc:1918] T 5d6a3fcfca8047dc94314c42adb241ac P 7173b2474809425497f7384daf982d2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:02:29.103016 4990 log.cc:1196] T 5d6a3fcfca8047dc94314c42adb241ac P 7173b2474809425497f7384daf982d2d: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/5d6a3fcfca8047dc94314c42adb241ac
I20260929 01:02:29.103188 4991 tablet_replica.cc:330] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde: stopping tablet replica
I20260929 01:02:29.103331 4990 ts_tablet_manager.cc:1939] T 5d6a3fcfca8047dc94314c42adb241ac P 7173b2474809425497f7384daf982d2d: Deleting consensus metadata
I20260929 01:02:29.103686 4991 raft_consensus.cc:2229] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:02:29.104175 4991 raft_consensus.cc:2258] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:29.104871 4478 catalog_manager.cc:4841] TS 7173b2474809425497f7384daf982d2d (127.0.98.3:34091): tablet 5d6a3fcfca8047dc94314c42adb241ac (table test-table [id=7141766042bb42bfbaacf8891b64a052]) successfully deleted
I20260929 01:02:29.105609 4991 ts_tablet_manager.cc:1905] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 01:02:29.108551 4479 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46321 (error 108)
I20260929 01:02:29.108788 4991 ts_tablet_manager.cc:1918] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:02:29.108882 4991 log.cc:1196] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790643720886468-392-0/minicluster-data/ts-1/wal/wals/5d6a3fcfca8047dc94314c42adb241ac
I20260929 01:02:29.109218 4991 ts_tablet_manager.cc:1939] T 5d6a3fcfca8047dc94314c42adb241ac P 12c2a71b7f1c4f84989af10986bdfcde: Deleting consensus metadata
I20260929 01:02:29.109387 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 4689
I20260929 01:02:29.109719 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.109372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46321 (local address 127.0.0.1:53062)
0929 01:02:29.109573 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:29.109635 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260929 01:02:29.110090 4479 catalog_manager.cc:4584] TS b5dd370dfa2b4c9b993c4c82e65b28e5 (127.0.98.1:46321): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5d6a3fcfca8047dc94314c42adb241ac: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46321: connect: Connection refused (error 111)
I20260929 01:02:29.115744 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 4822
I20260929 01:02:29.116200 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.115900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44215 (local address 127.0.0.1:37804)
0929 01:02:29.116065 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:29.116117 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 01:02:29.121147 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 4463
2026-09-29T01:02:29Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2368 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T01:02:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:02:29Z Disabled control of system clock
I20260929 01:02:29.150903 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42743
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 01:02:29.256953 4997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:29.257205 4997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:29.257241 4997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:29.261093 4997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:02:29.261149 4997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:29.261164 4997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:29.261181 4997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:02:29.261196 4997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:02:29.265920 4997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42743
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:29.266986 4997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:29.268299 4997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:29.276533 5005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:29.276525 5003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:29.276818 4997 server_base.cc:1034] running on GCE node
W20260929 01:02:29.276525 5002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:29.278060 4997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:29.279165 4997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:29.280316 4997 hybrid_clock.cc:648] HybridClock initialized: now 1790643749280286 us; error 45 us; skew 500 ppm
I20260929 01:02:29.282008 4997 webserver.cc:460] Webserver started at http://127.0.98.62:43293/ using document root <none> and password file <none>
I20260929 01:02:29.282584 4997 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:29.282644 4997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:29.282929 4997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:29.285013 4997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "ac11b588078a481a8a5fc391b4d4f000"
format_stamp: "Formatted at 2026-09-29 01:02:29 on dist-test-slave-7zdl"
server_key: "b408cadcf8325abf26e039a2c323ce42"
server_key_iv: "9a48f2a852a534f89af9cbb674cbf0c4"
server_key_version: "encryptionkey@0"
I20260929 01:02:29.285588 4997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "ac11b588078a481a8a5fc391b4d4f000"
format_stamp: "Formatted at 2026-09-29 01:02:29 on dist-test-slave-7zdl"
server_key: "b408cadcf8325abf26e039a2c323ce42"
server_key_iv: "9a48f2a852a534f89af9cbb674cbf0c4"
server_key_version: "encryptionkey@0"
I20260929 01:02:29.289130 4997 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 01:02:29.291364 5011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:29.292503 4997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:02:29.292615 4997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "ac11b588078a481a8a5fc391b4d4f000"
format_stamp: "Formatted at 2026-09-29 01:02:29 on dist-test-slave-7zdl"
server_key: "b408cadcf8325abf26e039a2c323ce42"
server_key_iv: "9a48f2a852a534f89af9cbb674cbf0c4"
server_key_version: "encryptionkey@0"
I20260929 01:02:29.292714 4997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:29.297187 4997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:29.297742 4997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:29.297920 4997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:29.304595 4997 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42743
I20260929 01:02:29.304607 5063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42743 every 8 connection(s)
I20260929 01:02:29.305462 4997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:02:29.305732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 4997
I20260929 01:02:29.305845 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:02:29.306150 392 external_mini_cluster.cc:1428] Setting key 9e22e0f6d21870950cca1388e909e468
I20260929 01:02:29.308562 5064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:29.311800 5067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.307457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38744 (local address 127.0.98.62:42743)
0929 01:02:29.308467 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:29.308480 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:29.308514 (+ 34us) server_negotiation.cc:407] Connection header received
0929 01:02:29.309158 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:29.309174 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:29.309497 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:29.309768 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:29.310563 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:29.310568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:29.310578 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:02:29.310587 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:29.310638 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:02:29.310653 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:29.310757 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:29.310932 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 01:02:29.311174 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":542,"threads_started":1}
I20260929 01:02:29.314548 5064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac11b588078a481a8a5fc391b4d4f000: Bootstrap starting.
I20260929 01:02:29.316663 5064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac11b588078a481a8a5fc391b4d4f000: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:29.317375 5064 log.cc:824] T 00000000000000000000000000000000 P ac11b588078a481a8a5fc391b4d4f000: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:29.319151 5064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac11b588078a481a8a5fc391b4d4f000: No bootstrap required, opened a new log
I20260929 01:02:29.322419 5064 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac11b588078a481a8a5fc391b4d4f000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac11b588078a481a8a5fc391b4d4f000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } }
I20260929 01:02:29.322667 5064 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac11b588078a481a8a5fc391b4d4f000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:29.322738 5064 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac11b588078a481a8a5fc391b4d4f000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac11b588078a481a8a5fc391b4d4f000, State: Initialized, Role: FOLLOWER
I20260929 01:02:29.323186 5064 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac11b588078a481a8a5fc391b4d4f000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac11b588078a481a8a5fc391b4d4f000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } }
I20260929 01:02:29.323329 5064 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac11b588078a481a8a5fc391b4d4f000 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:02:29.323408 5064 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac11b588078a481a8a5fc391b4d4f000 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:02:29.323511 5064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac11b588078a481a8a5fc391b4d4f000 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:29.324354 5064 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac11b588078a481a8a5fc391b4d4f000 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac11b588078a481a8a5fc391b4d4f000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } }
I20260929 01:02:29.324681 5064 leader_election.cc:302] T 00000000000000000000000000000000 P ac11b588078a481a8a5fc391b4d4f000 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac11b588078a481a8a5fc391b4d4f000; no voters:
I20260929 01:02:29.324959 5064 leader_election.cc:288] T 00000000000000000000000000000000 P ac11b588078a481a8a5fc391b4d4f000 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:02:29.325109 5069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac11b588078a481a8a5fc391b4d4f000 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:29.325357 5069 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac11b588078a481a8a5fc391b4d4f000 [term 1 LEADER]: Becoming Leader. State: Replica: ac11b588078a481a8a5fc391b4d4f000, State: Running, Role: LEADER
I20260929 01:02:29.325676 5069 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac11b588078a481a8a5fc391b4d4f000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac11b588078a481a8a5fc391b4d4f000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } }
I20260929 01:02:29.326161 5064 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac11b588078a481a8a5fc391b4d4f000 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:02:29.327378 5070 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac11b588078a481a8a5fc391b4d4f000 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac11b588078a481a8a5fc391b4d4f000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac11b588078a481a8a5fc391b4d4f000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } }
I20260929 01:02:29.327504 5070 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac11b588078a481a8a5fc391b4d4f000 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:29.327756 5071 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac11b588078a481a8a5fc391b4d4f000 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac11b588078a481a8a5fc391b4d4f000. Latest consensus state: current_term: 1 leader_uuid: "ac11b588078a481a8a5fc391b4d4f000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac11b588078a481a8a5fc391b4d4f000" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } }
I20260929 01:02:29.327913 5071 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac11b588078a481a8a5fc391b4d4f000 [sys.catalog]: This master's current role is: LEADER
W20260929 01:02:29.330890 5084 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac11b588078a481a8a5fc391b4d4f000: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:02:29.330953 5084 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:02:29.331027 5078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:02:29.331718 5078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:02:29.337137 5078 catalog_manager.cc:1347] Generated new cluster ID: b1e99ea32f604ec2be43d288a90ae3a0
I20260929 01:02:29.337205 5078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:02:29.345688 5078 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:02:29.345949 5078 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:02:29.355003 5078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac11b588078a481a8a5fc391b4d4f000: Generated new TSK 0
I20260929 01:02:29.355628 5078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:02:29.362038 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42743
--builtin_ntp_servers=127.0.98.20:40687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 01:02:29.470121 5088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:29.470364 5088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:29.470402 5088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:29.474484 5088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:29.474534 5088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:29.474586 5088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:02:29.479357 5088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:29.480291 5088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:29.481559 5088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:29.490705 5094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:29.490820 5093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:29.490705 5096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:29.491073 5088 server_base.cc:1034] running on GCE node
I20260929 01:02:29.491490 5088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:29.492110 5088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:29.493280 5088 hybrid_clock.cc:648] HybridClock initialized: now 1790643749493264 us; error 27 us; skew 500 ppm
I20260929 01:02:29.495042 5088 webserver.cc:460] Webserver started at http://127.0.98.1:42057/ using document root <none> and password file <none>
I20260929 01:02:29.495635 5088 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:29.495720 5088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:29.495925 5088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:29.497998 5088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "c4fda1724cb14b4ca83e1a14b74c0f0c"
format_stamp: "Formatted at 2026-09-29 01:02:29 on dist-test-slave-7zdl"
server_key: "103cd9aabca995c8b59602b471acd6a0"
server_key_iv: "278c95bfbf8566b80da8d98773d1afa5"
server_key_version: "encryptionkey@0"
I20260929 01:02:29.498534 5088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c4fda1724cb14b4ca83e1a14b74c0f0c"
format_stamp: "Formatted at 2026-09-29 01:02:29 on dist-test-slave-7zdl"
server_key: "103cd9aabca995c8b59602b471acd6a0"
server_key_iv: "278c95bfbf8566b80da8d98773d1afa5"
server_key_version: "encryptionkey@0"
I20260929 01:02:29.502055 5088 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 01:02:29.504351 5102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:29.505479 5088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:02:29.505606 5088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "c4fda1724cb14b4ca83e1a14b74c0f0c"
format_stamp: "Formatted at 2026-09-29 01:02:29 on dist-test-slave-7zdl"
server_key: "103cd9aabca995c8b59602b471acd6a0"
server_key_iv: "278c95bfbf8566b80da8d98773d1afa5"
server_key_version: "encryptionkey@0"
I20260929 01:02:29.505708 5088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:29.510116 5088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:29.510663 5088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:29.510890 5088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:29.511405 5088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:29.512243 5088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:29.512310 5088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:29.512372 5088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:29.512422 5088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:29.520773 5088 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36187
I20260929 01:02:29.520802 5215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36187 every 8 connection(s)
I20260929 01:02:29.521611 5088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:02:29.527252 5067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.523321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53761 (local address 127.0.98.62:42743)
0929 01:02:29.523447 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:29.523450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:29.524427 (+ 977us) server_negotiation.cc:407] Connection header received
0929 01:02:29.525631 (+ 1204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:29.525634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:29.525671 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:29.525744 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:29.526868 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:29.526870 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:29.526872 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:02:29.526882 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:29.526899 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:02:29.526907 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:29.527015 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:29.527115 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 01:02:29.527151 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 01:02:29.527578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 5088
I20260929 01:02:29.527705 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:02:29.528074 392 external_mini_cluster.cc:1428] Setting key 3a16f3809683bfe29fbc289e5b86fc8a
I20260929 01:02:29.528355 5218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.523670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42743 (local address 127.0.98.1:53761)
0929 01:02:29.524297 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:29.524335 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:29.525455 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:29.525892 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:29.525898 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:29.525982 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:29.526574 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:02:29.526575 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:29.526614 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:02:29.526696 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:29.526998 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:29.527005 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:29.527009 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 01:02:29.527138 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 01:02:29.527317 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":120,"threads_started":1}
I20260929 01:02:29.529551 5216 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42743
I20260929 01:02:29.529773 5216 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:29.530140 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42743
--builtin_ntp_servers=127.0.98.20:40687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 01:02:29.530218 5216 heartbeater.cc:511] Master 127.0.98.62:42743 requested a full tablet report, sending...
I20260929 01:02:29.531713 5028 ts_manager.cc:194] Registered new tserver with Master: c4fda1724cb14b4ca83e1a14b74c0f0c (127.0.98.1:36187)
W20260929 01:02:29.637549 5219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:29.637873 5219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:29.637930 5219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:29.642601 5219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:29.642655 5219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:29.642710 5219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:02:29.647517 5219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:29.648511 5219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:29.649808 5219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:29.659169 5227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:29.659215 5219 server_base.cc:1034] running on GCE node
W20260929 01:02:29.659420 5224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:29.659397 5225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:29.659896 5219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:29.660482 5219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:29.661652 5219 hybrid_clock.cc:648] HybridClock initialized: now 1790643749661638 us; error 35 us; skew 500 ppm
I20260929 01:02:29.663417 5219 webserver.cc:460] Webserver started at http://127.0.98.2:45521/ using document root <none> and password file <none>
I20260929 01:02:29.663962 5219 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:29.664062 5219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:29.664274 5219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:29.666253 5219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "e329ae58193244039a82a46fc118983c"
format_stamp: "Formatted at 2026-09-29 01:02:29 on dist-test-slave-7zdl"
server_key: "5244391f9b56de8b7bc1598b2041bfce"
server_key_iv: "4d3ffc4e3d88d08f26ca5858cfe57c37"
server_key_version: "encryptionkey@0"
I20260929 01:02:29.666851 5219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e329ae58193244039a82a46fc118983c"
format_stamp: "Formatted at 2026-09-29 01:02:29 on dist-test-slave-7zdl"
server_key: "5244391f9b56de8b7bc1598b2041bfce"
server_key_iv: "4d3ffc4e3d88d08f26ca5858cfe57c37"
server_key_version: "encryptionkey@0"
I20260929 01:02:29.670321 5219 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:02:29.673017 5233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:29.674186 5219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 01:02:29.674324 5219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "e329ae58193244039a82a46fc118983c"
format_stamp: "Formatted at 2026-09-29 01:02:29 on dist-test-slave-7zdl"
server_key: "5244391f9b56de8b7bc1598b2041bfce"
server_key_iv: "4d3ffc4e3d88d08f26ca5858cfe57c37"
server_key_version: "encryptionkey@0"
I20260929 01:02:29.674431 5219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:29.678965 5219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:29.679579 5219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:29.679747 5219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:29.680260 5219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:29.681130 5219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:29.681195 5219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:29.681258 5219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:29.681304 5219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:29.689750 5219 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39667
I20260929 01:02:29.689774 5346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39667 every 8 connection(s)
I20260929 01:02:29.690579 5219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:02:29.695833 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 5219
I20260929 01:02:29.695966 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:02:29.696255 392 external_mini_cluster.cc:1428] Setting key 786e1335b17cf4a151eb73a10a6b95e4
I20260929 01:02:29.696769 5067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.692253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43711 (local address 127.0.98.62:42743)
0929 01:02:29.692382 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:29.692386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:29.693520 (+ 1134us) server_negotiation.cc:407] Connection header received
0929 01:02:29.694890 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:29.694894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:29.694941 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:29.695035 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:29.696273 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:29.696278 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:29.696281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:02:29.696295 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:29.696319 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:02:29.696333 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:29.696425 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:29.696567 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 01:02:29.696614 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 01:02:29.697710 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.692639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42743 (local address 127.0.98.2:43711)
0929 01:02:29.693361 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:29.693403 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:29.694617 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:29.695198 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:29.695205 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:29.695332 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:29.695969 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:02:29.695973 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:29.696047 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:02:29.696127 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:29.696418 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:29.696426 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:29.696433 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 01:02:29.696609 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 01:02:29.696835 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":116,"threads_started":1}
I20260929 01:02:29.698208 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42743
--builtin_ntp_servers=127.0.98.20:40687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 01:02:29.698892 5347 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42743
I20260929 01:02:29.699090 5347 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:29.699458 5347 heartbeater.cc:511] Master 127.0.98.62:42743 requested a full tablet report, sending...
I20260929 01:02:29.700449 5028 ts_manager.cc:194] Registered new tserver with Master: e329ae58193244039a82a46fc118983c (127.0.98.2:39667)
W20260929 01:02:29.814908 5350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:29.815167 5350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:29.815243 5350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:29.819180 5350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:29.819227 5350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:29.819278 5350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:02:29.823834 5350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:29.824823 5350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:29.826202 5350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:29.835886 5355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:29.835857 5356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:29.835866 5358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:29.836355 5350 server_base.cc:1034] running on GCE node
I20260929 01:02:29.836743 5350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:29.837337 5350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:29.838519 5350 hybrid_clock.cc:648] HybridClock initialized: now 1790643749838493 us; error 38 us; skew 500 ppm
I20260929 01:02:29.840345 5350 webserver.cc:460] Webserver started at http://127.0.98.3:34325/ using document root <none> and password file <none>
I20260929 01:02:29.840945 5350 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:29.841032 5350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:29.841248 5350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:29.843313 5350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "7aee4e028ff34d57b5b8b0c242fdc5df"
format_stamp: "Formatted at 2026-09-29 01:02:29 on dist-test-slave-7zdl"
server_key: "01ac0853290363aa776193045046afb0"
server_key_iv: "f341afc462d41f08a43f76e0b6800651"
server_key_version: "encryptionkey@0"
I20260929 01:02:29.843849 5350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7aee4e028ff34d57b5b8b0c242fdc5df"
format_stamp: "Formatted at 2026-09-29 01:02:29 on dist-test-slave-7zdl"
server_key: "01ac0853290363aa776193045046afb0"
server_key_iv: "f341afc462d41f08a43f76e0b6800651"
server_key_version: "encryptionkey@0"
I20260929 01:02:29.847464 5350 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:02:29.849715 5364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:29.850901 5350 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:02:29.851024 5350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "7aee4e028ff34d57b5b8b0c242fdc5df"
format_stamp: "Formatted at 2026-09-29 01:02:29 on dist-test-slave-7zdl"
server_key: "01ac0853290363aa776193045046afb0"
server_key_iv: "f341afc462d41f08a43f76e0b6800651"
server_key_version: "encryptionkey@0"
I20260929 01:02:29.851130 5350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:29.855664 5350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:29.856259 5350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:29.856385 5350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:29.856868 5350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:29.857651 5350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:29.857699 5350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:29.857733 5350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:29.857754 5350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:29.865933 5350 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36217
I20260929 01:02:29.865964 5477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36217 every 8 connection(s)
I20260929 01:02:29.866837 5350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:02:29.872574 5067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.868593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46799 (local address 127.0.98.62:42743)
0929 01:02:29.868716 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:29.868720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:29.869664 (+ 944us) server_negotiation.cc:407] Connection header received
0929 01:02:29.870934 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:29.870938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:29.870980 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:29.871080 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:29.872117 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:29.872120 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:29.872123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:02:29.872134 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:29.872153 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:02:29.872163 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:29.872254 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:29.872393 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 01:02:29.872442 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 01:02:29.873309 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.868920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42743 (local address 127.0.98.3:46799)
0929 01:02:29.869529 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:29.869566 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:29.870701 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:29.871226 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:29.871233 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:29.871314 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:29.871876 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:02:29.871878 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:29.871919 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:02:29.871981 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:29.872250 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:29.872259 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:29.872266 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 01:02:29.872424 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 01:02:29.872637 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":113,"threads_started":1}
I20260929 01:02:29.874306 5478 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42743
I20260929 01:02:29.874513 5478 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:29.874928 5478 heartbeater.cc:511] Master 127.0.98.62:42743 requested a full tablet report, sending...
I20260929 01:02:29.875847 5028 ts_manager.cc:194] Registered new tserver with Master: 7aee4e028ff34d57b5b8b0c242fdc5df (127.0.98.3:36217)
I20260929 01:02:29.876418 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 5350
I20260929 01:02:29.876550 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:02:29.876829 392 external_mini_cluster.cc:1428] Setting key 2b862279032949805d4bb92e7a6c859a
I20260929 01:02:29.878710 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:02:29.881901 5067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.880497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38746 (local address 127.0.98.62:42743)
0929 01:02:29.880622 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:29.880626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:29.880756 (+ 130us) server_negotiation.cc:407] Connection header received
0929 01:02:29.880840 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:29.880843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:29.880870 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:29.880934 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:29.881591 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:29.881593 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:29.881595 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:02:29.881602 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:29.881616 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:02:29.881623 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:29.881682 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:29.881723 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 01:02:29.881752 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 01:02:29.884558 5028 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:02:29.887579 5028 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:02:29.898857 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.895219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36217 (local address 127.0.0.1:39724)
0929 01:02:29.896186 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:29.896238 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:29.896430 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:29.897321 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:29.897327 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:29.897399 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:29.898001 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:02:29.898003 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:29.898018 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:02:29.898102 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:29.898482 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:29.898490 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:29.898499 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 01:02:29.898628 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 01:02:29.898694 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":78,"threads_started":1}
I20260929 01:02:29.898904 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.895360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39724 (local address 127.0.98.3:36217)
0929 01:02:29.895707 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:29.895718 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:29.896328 (+ 610us) server_negotiation.cc:407] Connection header received
0929 01:02:29.896573 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:29.896579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:29.896765 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:29.896936 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:29.898238 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:29.898245 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:29.898260 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:02:29.898276 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:29.898326 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:02:29.898352 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:29.898488 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:29.898642 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 01:02:29.898713 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":158,"threads_started":1}
I20260929 01:02:29.899458 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.896049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57110 (local address 127.0.98.2:39667)
0929 01:02:29.896594 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:29.896612 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:29.896864 (+ 252us) server_negotiation.cc:407] Connection header received
0929 01:02:29.896999 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:29.897005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:29.897181 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:29.897310 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:29.898552 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:29.898558 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:29.898569 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:02:29.898586 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:29.898620 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:02:29.898645 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:29.898766 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:29.899214 (+ 448us) server_negotiation.cc:299] Negotiation successful
0929 01:02:29.899284 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":129,"threads_started":1}
I20260929 01:02:29.899123 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.895349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39667 (local address 127.0.0.1:57110)
0929 01:02:29.896739 (+ 1390us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:29.896752 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:29.896898 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:29.897547 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:29.897550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:29.897566 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:29.898367 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:02:29.898369 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:29.898380 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:02:29.898409 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:29.898869 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:29.898872 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:29.898875 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 01:02:29.898956 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 01:02:29.898995 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1333,"thread_start_us":79,"threads_started":1}
I20260929 01:02:29.899889 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.895970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36187 (local address 127.0.0.1:47086)
0929 01:02:29.897089 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:29.897101 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:29.897169 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:29.898108 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:29.898111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:29.898129 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:29.899247 (+ 1118us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:02:29.899251 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:29.899264 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:02:29.899302 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:29.899696 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:29.899699 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:29.899702 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 01:02:29.899765 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 01:02:29.899799 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"thread_start_us":90,"threads_started":1}
I20260929 01:02:29.900381 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.896100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47086 (local address 127.0.98.1:36187)
0929 01:02:29.896501 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:29.896517 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:29.897185 (+ 668us) server_negotiation.cc:407] Connection header received
0929 01:02:29.897253 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:29.897259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:29.897429 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:29.897558 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:29.899411 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:29.899419 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:29.899430 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:02:29.899443 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:29.899493 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:02:29.899540 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:29.899655 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:29.900148 (+ 493us) server_negotiation.cc:299] Negotiation successful
0929 01:02:29.900222 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":134,"threads_started":1}
I20260929 01:02:29.902081 5281 tablet_service.cc:1467] Processing CreateTablet for tablet e80a315ea76c4561b3fdff5e9177de2f (DEFAULT_TABLE table=test-table [id=e480f5bd12b4421da97535998167a931]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:29.902752 5150 tablet_service.cc:1467] Processing CreateTablet for tablet e80a315ea76c4561b3fdff5e9177de2f (DEFAULT_TABLE table=test-table [id=e480f5bd12b4421da97535998167a931]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:29.903249 5281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e80a315ea76c4561b3fdff5e9177de2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:29.903644 5150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e80a315ea76c4561b3fdff5e9177de2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:29.904624 5412 tablet_service.cc:1467] Processing CreateTablet for tablet e80a315ea76c4561b3fdff5e9177de2f (DEFAULT_TABLE table=test-table [id=e480f5bd12b4421da97535998167a931]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:29.905570 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e80a315ea76c4561b3fdff5e9177de2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:29.909535 5495 tablet_bootstrap.cc:492] T e80a315ea76c4561b3fdff5e9177de2f P c4fda1724cb14b4ca83e1a14b74c0f0c: Bootstrap starting.
I20260929 01:02:29.911121 5496 tablet_bootstrap.cc:492] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c: Bootstrap starting.
I20260929 01:02:29.912056 5497 tablet_bootstrap.cc:492] T e80a315ea76c4561b3fdff5e9177de2f P 7aee4e028ff34d57b5b8b0c242fdc5df: Bootstrap starting.
I20260929 01:02:29.912580 5495 tablet_bootstrap.cc:654] T e80a315ea76c4561b3fdff5e9177de2f P c4fda1724cb14b4ca83e1a14b74c0f0c: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:29.913060 5496 tablet_bootstrap.cc:654] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:29.913482 5495 log.cc:824] T e80a315ea76c4561b3fdff5e9177de2f P c4fda1724cb14b4ca83e1a14b74c0f0c: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:29.913774 5496 log.cc:824] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:29.914031 5497 tablet_bootstrap.cc:654] T e80a315ea76c4561b3fdff5e9177de2f P 7aee4e028ff34d57b5b8b0c242fdc5df: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:29.914773 5497 log.cc:824] T e80a315ea76c4561b3fdff5e9177de2f P 7aee4e028ff34d57b5b8b0c242fdc5df: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:29.915535 5495 tablet_bootstrap.cc:492] T e80a315ea76c4561b3fdff5e9177de2f P c4fda1724cb14b4ca83e1a14b74c0f0c: No bootstrap required, opened a new log
I20260929 01:02:29.915608 5496 tablet_bootstrap.cc:492] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c: No bootstrap required, opened a new log
I20260929 01:02:29.915722 5495 ts_tablet_manager.cc:1397] T e80a315ea76c4561b3fdff5e9177de2f P c4fda1724cb14b4ca83e1a14b74c0f0c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 01:02:29.915735 5496 ts_tablet_manager.cc:1397] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 01:02:29.916561 5497 tablet_bootstrap.cc:492] T e80a315ea76c4561b3fdff5e9177de2f P 7aee4e028ff34d57b5b8b0c242fdc5df: No bootstrap required, opened a new log
I20260929 01:02:29.916735 5497 ts_tablet_manager.cc:1397] T e80a315ea76c4561b3fdff5e9177de2f P 7aee4e028ff34d57b5b8b0c242fdc5df: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 01:02:29.919930 5497 raft_consensus.cc:348] T e80a315ea76c4561b3fdff5e9177de2f P 7aee4e028ff34d57b5b8b0c242fdc5df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aee4e028ff34d57b5b8b0c242fdc5df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36217 } } peers { permanent_uuid: "e329ae58193244039a82a46fc118983c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39667 } } peers { permanent_uuid: "c4fda1724cb14b4ca83e1a14b74c0f0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36187 } }
I20260929 01:02:29.920164 5497 raft_consensus.cc:374] T e80a315ea76c4561b3fdff5e9177de2f P 7aee4e028ff34d57b5b8b0c242fdc5df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:29.920217 5497 raft_consensus.cc:729] T e80a315ea76c4561b3fdff5e9177de2f P 7aee4e028ff34d57b5b8b0c242fdc5df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aee4e028ff34d57b5b8b0c242fdc5df, State: Initialized, Role: FOLLOWER
I20260929 01:02:29.920243 5496 raft_consensus.cc:348] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aee4e028ff34d57b5b8b0c242fdc5df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36217 } } peers { permanent_uuid: "e329ae58193244039a82a46fc118983c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39667 } } peers { permanent_uuid: "c4fda1724cb14b4ca83e1a14b74c0f0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36187 } }
I20260929 01:02:29.920243 5495 raft_consensus.cc:348] T e80a315ea76c4561b3fdff5e9177de2f P c4fda1724cb14b4ca83e1a14b74c0f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aee4e028ff34d57b5b8b0c242fdc5df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36217 } } peers { permanent_uuid: "e329ae58193244039a82a46fc118983c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39667 } } peers { permanent_uuid: "c4fda1724cb14b4ca83e1a14b74c0f0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36187 } }
I20260929 01:02:29.920482 5496 raft_consensus.cc:374] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:29.920482 5495 raft_consensus.cc:374] T e80a315ea76c4561b3fdff5e9177de2f P c4fda1724cb14b4ca83e1a14b74c0f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:29.920537 5496 raft_consensus.cc:729] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e329ae58193244039a82a46fc118983c, State: Initialized, Role: FOLLOWER
I20260929 01:02:29.920544 5495 raft_consensus.cc:729] T e80a315ea76c4561b3fdff5e9177de2f P c4fda1724cb14b4ca83e1a14b74c0f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4fda1724cb14b4ca83e1a14b74c0f0c, State: Initialized, Role: FOLLOWER
I20260929 01:02:29.920667 5497 consensus_queue.cc:260] T e80a315ea76c4561b3fdff5e9177de2f P 7aee4e028ff34d57b5b8b0c242fdc5df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aee4e028ff34d57b5b8b0c242fdc5df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36217 } } peers { permanent_uuid: "e329ae58193244039a82a46fc118983c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39667 } } peers { permanent_uuid: "c4fda1724cb14b4ca83e1a14b74c0f0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36187 } }
I20260929 01:02:29.920961 5495 consensus_queue.cc:260] T e80a315ea76c4561b3fdff5e9177de2f P c4fda1724cb14b4ca83e1a14b74c0f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aee4e028ff34d57b5b8b0c242fdc5df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36217 } } peers { permanent_uuid: "e329ae58193244039a82a46fc118983c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39667 } } peers { permanent_uuid: "c4fda1724cb14b4ca83e1a14b74c0f0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36187 } }
I20260929 01:02:29.920961 5496 consensus_queue.cc:260] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aee4e028ff34d57b5b8b0c242fdc5df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36217 } } peers { permanent_uuid: "e329ae58193244039a82a46fc118983c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39667 } } peers { permanent_uuid: "c4fda1724cb14b4ca83e1a14b74c0f0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36187 } }
I20260929 01:02:29.921345 5478 heartbeater.cc:503] Master 127.0.98.62:42743 was elected leader, sending a full tablet report...
I20260929 01:02:29.921653 5497 ts_tablet_manager.cc:1428] T e80a315ea76c4561b3fdff5e9177de2f P 7aee4e028ff34d57b5b8b0c242fdc5df: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260929 01:02:29.921860 5496 ts_tablet_manager.cc:1428] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260929 01:02:29.921860 5495 ts_tablet_manager.cc:1428] T e80a315ea76c4561b3fdff5e9177de2f P c4fda1724cb14b4ca83e1a14b74c0f0c: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260929 01:02:29.922181 5216 heartbeater.cc:503] Master 127.0.98.62:42743 was elected leader, sending a full tablet report...
I20260929 01:02:29.922947 5347 heartbeater.cc:503] Master 127.0.98.62:42743 was elected leader, sending a full tablet report...
W20260929 01:02:29.941560 5348 tablet.cc:2369] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:02:29.967445 5502 raft_consensus.cc:482] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:02:29.967657 5502 raft_consensus.cc:504] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aee4e028ff34d57b5b8b0c242fdc5df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36217 } } peers { permanent_uuid: "e329ae58193244039a82a46fc118983c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39667 } } peers { permanent_uuid: "c4fda1724cb14b4ca83e1a14b74c0f0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36187 } }
I20260929 01:02:29.968778 5502 leader_election.cc:288] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7aee4e028ff34d57b5b8b0c242fdc5df (127.0.98.3:36217), c4fda1724cb14b4ca83e1a14b74c0f0c (127.0.98.1:36187)
I20260929 01:02:29.970876 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.968978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36217 (local address 127.0.98.2:55795)
0929 01:02:29.969107 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:29.969122 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:29.969202 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:29.969574 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:29.969580 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:29.969603 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:29.970244 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:02:29.970247 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:29.970260 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:02:29.970295 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:29.970586 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:29.970589 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:29.970593 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 01:02:29.970666 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 01:02:29.970695 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 01:02:29.970890 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.969167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55795 (local address 127.0.98.3:36217)
0929 01:02:29.969310 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:29.969315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:29.969329 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:02:29.969372 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:29.969375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:29.969414 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:29.969502 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:29.970435 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:29.970438 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:29.970440 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:02:29.970464 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:29.970489 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:02:29.970500 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:29.970594 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:29.970677 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 01:02:29.970722 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 01:02:29.971007 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.969147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54705 (local address 127.0.98.1:36187)
0929 01:02:29.969272 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:29.969277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:29.969567 (+ 290us) server_negotiation.cc:407] Connection header received
0929 01:02:29.969724 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:29.969728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:29.969776 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:29.969856 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:29.970616 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:29.970619 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:29.970622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:02:29.970632 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:29.970654 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:02:29.970665 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:29.970745 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:29.970837 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 01:02:29.970878 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 01:02:29.971213 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.969021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36187 (local address 127.0.98.2:54705)
0929 01:02:29.969462 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:29.969478 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:29.969587 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:29.969893 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:29.969897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:29.969912 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:29.970451 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:02:29.970453 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:29.970465 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:02:29.970495 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:29.970741 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:29.970743 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:29.970746 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 01:02:29.970839 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 01:02:29.970867 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":13,"thread_start_us":132,"threads_started":1}
I20260929 01:02:29.971592 5432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e80a315ea76c4561b3fdff5e9177de2f" candidate_uuid: "e329ae58193244039a82a46fc118983c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aee4e028ff34d57b5b8b0c242fdc5df" is_pre_election: true
I20260929 01:02:29.971761 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e80a315ea76c4561b3fdff5e9177de2f" candidate_uuid: "e329ae58193244039a82a46fc118983c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4fda1724cb14b4ca83e1a14b74c0f0c" is_pre_election: true
I20260929 01:02:29.971879 5432 raft_consensus.cc:2454] T e80a315ea76c4561b3fdff5e9177de2f P 7aee4e028ff34d57b5b8b0c242fdc5df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e329ae58193244039a82a46fc118983c in term 0.
I20260929 01:02:29.972034 5170 raft_consensus.cc:2454] T e80a315ea76c4561b3fdff5e9177de2f P c4fda1724cb14b4ca83e1a14b74c0f0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e329ae58193244039a82a46fc118983c in term 0.
I20260929 01:02:29.972381 5234 leader_election.cc:302] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7aee4e028ff34d57b5b8b0c242fdc5df, c4fda1724cb14b4ca83e1a14b74c0f0c, e329ae58193244039a82a46fc118983c; no voters:
I20260929 01:02:29.972636 5502 raft_consensus.cc:2781] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:02:29.972723 5502 raft_consensus.cc:482] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:02:29.972781 5502 raft_consensus.cc:3037] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:29.973785 5502 raft_consensus.cc:504] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aee4e028ff34d57b5b8b0c242fdc5df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36217 } } peers { permanent_uuid: "e329ae58193244039a82a46fc118983c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39667 } } peers { permanent_uuid: "c4fda1724cb14b4ca83e1a14b74c0f0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36187 } }
I20260929 01:02:29.974093 5502 leader_election.cc:288] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c [CANDIDATE]: Term 1 election: Requested vote from peers 7aee4e028ff34d57b5b8b0c242fdc5df (127.0.98.3:36217), c4fda1724cb14b4ca83e1a14b74c0f0c (127.0.98.1:36187)
I20260929 01:02:29.974409 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e80a315ea76c4561b3fdff5e9177de2f" candidate_uuid: "e329ae58193244039a82a46fc118983c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4fda1724cb14b4ca83e1a14b74c0f0c"
I20260929 01:02:29.974524 5170 raft_consensus.cc:3037] T e80a315ea76c4561b3fdff5e9177de2f P c4fda1724cb14b4ca83e1a14b74c0f0c [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:29.974905 5432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e80a315ea76c4561b3fdff5e9177de2f" candidate_uuid: "e329ae58193244039a82a46fc118983c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aee4e028ff34d57b5b8b0c242fdc5df"
I20260929 01:02:29.974997 5432 raft_consensus.cc:3037] T e80a315ea76c4561b3fdff5e9177de2f P 7aee4e028ff34d57b5b8b0c242fdc5df [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:29.975665 5170 raft_consensus.cc:2454] T e80a315ea76c4561b3fdff5e9177de2f P c4fda1724cb14b4ca83e1a14b74c0f0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e329ae58193244039a82a46fc118983c in term 1.
I20260929 01:02:29.975943 5235 leader_election.cc:302] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4fda1724cb14b4ca83e1a14b74c0f0c, e329ae58193244039a82a46fc118983c; no voters:
I20260929 01:02:29.976135 5432 raft_consensus.cc:2454] T e80a315ea76c4561b3fdff5e9177de2f P 7aee4e028ff34d57b5b8b0c242fdc5df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e329ae58193244039a82a46fc118983c in term 1.
I20260929 01:02:29.976161 5502 raft_consensus.cc:2781] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:29.976387 5502 raft_consensus.cc:686] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c [term 1 LEADER]: Becoming Leader. State: Replica: e329ae58193244039a82a46fc118983c, State: Running, Role: LEADER
I20260929 01:02:29.976697 5502 consensus_queue.cc:237] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aee4e028ff34d57b5b8b0c242fdc5df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36217 } } peers { permanent_uuid: "e329ae58193244039a82a46fc118983c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39667 } } peers { permanent_uuid: "c4fda1724cb14b4ca83e1a14b74c0f0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36187 } }
I20260929 01:02:29.979447 5027 catalog_manager.cc:5495] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c reported cstate change: term changed from 0 to 1, leader changed from <none> to e329ae58193244039a82a46fc118983c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e329ae58193244039a82a46fc118983c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aee4e028ff34d57b5b8b0c242fdc5df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e329ae58193244039a82a46fc118983c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4fda1724cb14b4ca83e1a14b74c0f0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36187 } health_report { overall_health: UNKNOWN } } }
I20260929 01:02:29.995359 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:29.993957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57124 (local address 127.0.98.2:39667)
0929 01:02:29.994082 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:29.994086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:29.994098 (+ 12us) server_negotiation.cc:407] Connection header received
0929 01:02:29.994163 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:29.994165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:29.994217 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:29.994283 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:29.995086 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:29.995088 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:29.995090 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:02:29.995098 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:29.995114 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:02:29.995121 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:29.995188 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:29.995234 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 01:02:29.995265 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 01:02:30.001124 5432 raft_consensus.cc:1261] T e80a315ea76c4561b3fdff5e9177de2f P 7aee4e028ff34d57b5b8b0c242fdc5df [term 1 FOLLOWER]: Refusing update from remote peer e329ae58193244039a82a46fc118983c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:30.001154 5170 raft_consensus.cc:1261] T e80a315ea76c4561b3fdff5e9177de2f P c4fda1724cb14b4ca83e1a14b74c0f0c [term 1 FOLLOWER]: Refusing update from remote peer e329ae58193244039a82a46fc118983c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:30.001852 5505 consensus_queue.cc:1035] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aee4e028ff34d57b5b8b0c242fdc5df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:30.002013 5502 consensus_queue.cc:1035] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4fda1724cb14b4ca83e1a14b74c0f0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:30.017985 5026 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:30.018119 5026 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:30.020509 5026 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac11b588078a481a8a5fc391b4d4f000: Sending DeleteTablet for 3 replicas of tablet e80a315ea76c4561b3fdff5e9177de2f
I20260929 01:02:30.021152 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet e80a315ea76c4561b3fdff5e9177de2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:30 UTC) from {username='slave'} at 127.0.0.1:39724
I20260929 01:02:30.021201 5281 tablet_service.cc:1514] Processing DeleteTablet for tablet e80a315ea76c4561b3fdff5e9177de2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:30 UTC) from {username='slave'} at 127.0.0.1:57110
I20260929 01:02:30.021574 5518 tablet_replica.cc:330] T e80a315ea76c4561b3fdff5e9177de2f P 7aee4e028ff34d57b5b8b0c242fdc5df: stopping tablet replica
I20260929 01:02:30.021636 5150 tablet_service.cc:1514] Processing DeleteTablet for tablet e80a315ea76c4561b3fdff5e9177de2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:30 UTC) from {username='slave'} at 127.0.0.1:47086
I20260929 01:02:30.021782 5518 raft_consensus.cc:2229] T e80a315ea76c4561b3fdff5e9177de2f P 7aee4e028ff34d57b5b8b0c242fdc5df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:30.022032 5518 raft_consensus.cc:2258] T e80a315ea76c4561b3fdff5e9177de2f P 7aee4e028ff34d57b5b8b0c242fdc5df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:30.022053 5520 tablet_replica.cc:330] T e80a315ea76c4561b3fdff5e9177de2f P c4fda1724cb14b4ca83e1a14b74c0f0c: stopping tablet replica
I20260929 01:02:30.022431 5520 raft_consensus.cc:2229] T e80a315ea76c4561b3fdff5e9177de2f P c4fda1724cb14b4ca83e1a14b74c0f0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:30.022698 5520 raft_consensus.cc:2258] T e80a315ea76c4561b3fdff5e9177de2f P c4fda1724cb14b4ca83e1a14b74c0f0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:30.023058 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 5088
I20260929 01:02:30.023881 5518 ts_tablet_manager.cc:1905] T e80a315ea76c4561b3fdff5e9177de2f P 7aee4e028ff34d57b5b8b0c242fdc5df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:30.026317 5518 ts_tablet_manager.cc:1918] T e80a315ea76c4561b3fdff5e9177de2f P 7aee4e028ff34d57b5b8b0c242fdc5df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:02:30.026428 5518 log.cc:1196] T e80a315ea76c4561b3fdff5e9177de2f P 7aee4e028ff34d57b5b8b0c242fdc5df: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/e80a315ea76c4561b3fdff5e9177de2f
I20260929 01:02:30.026710 5518 ts_tablet_manager.cc:1939] T e80a315ea76c4561b3fdff5e9177de2f P 7aee4e028ff34d57b5b8b0c242fdc5df: Deleting consensus metadata
I20260929 01:02:30.027822 5012 catalog_manager.cc:4841] TS 7aee4e028ff34d57b5b8b0c242fdc5df (127.0.98.3:36217): tablet e80a315ea76c4561b3fdff5e9177de2f (table test-table [id=e480f5bd12b4421da97535998167a931]) successfully deleted
I20260929 01:02:30.031003 5519 tablet_replica.cc:330] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c: stopping tablet replica
I20260929 01:02:30.031278 5519 raft_consensus.cc:2229] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c [term 1 LEADER]: Raft consensus shutting down.
W20260929 01:02:30.031518 5013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36187 (error 108)
I20260929 01:02:30.031679 5519 raft_consensus.cc:2258] T e80a315ea76c4561b3fdff5e9177de2f P e329ae58193244039a82a46fc118983c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:30.031720 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 5219
I20260929 01:02:30.033660 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:30.033306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36187 (local address 127.0.0.1:47102)
0929 01:02:30.033462 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:30.033554 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 01:02:30.033845 5013 catalog_manager.cc:4584] TS c4fda1724cb14b4ca83e1a14b74c0f0c (127.0.98.1:36187): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e80a315ea76c4561b3fdff5e9177de2f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36187: connect: Connection refused (error 111)
I20260929 01:02:30.037683 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 5350
I20260929 01:02:30.037830 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:30.037544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39667 (local address 127.0.0.1:57132)
0929 01:02:30.037680 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:30.037741 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 01:02:30.042935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 4997
2026-09-29T01:02:30Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (920 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 01:02:30.064142 392 mini_oidc.cc:152] Starting JWKS server
I20260929 01:02:30.065651 392 webserver.cc:460] Webserver started at https://127.0.0.1:43769/ using document root <none> and password file <none>
I20260929 01:02:30.065825 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 01:02:30.066083 392 webserver.cc:460] Webserver started at http://127.0.0.1:33529/ using document root <none> and password file <none>
2026-09-29T01:02:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:02:30Z Disabled control of system clock
I20260929 01:02:30.074225 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36629
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36629
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43769/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643720886468-392-0/testchainca.cert with env {}
W20260929 01:02:30.181638 5527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:30.181876 5527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:30.181952 5527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:30.186038 5527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:02:30.186087 5527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:30.186103 5527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:30.186151 5527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:02:30.186178 5527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 01:02:30.186213 5527 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 01:02:30.186264 5527 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43769/jwks.json
I20260929 01:02:30.191469 5527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36629
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43769/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643720886468-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:30.192420 5527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:30.193377 5527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:30.201285 5532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:30.201498 5533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:30.201610 5535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:30.201474 5527 server_base.cc:1034] running on GCE node
I20260929 01:02:30.202157 5527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:30.203235 5527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:30.204494 5527 hybrid_clock.cc:648] HybridClock initialized: now 1790643750204486 us; error 28 us; skew 500 ppm
I20260929 01:02:30.206318 5527 webserver.cc:460] Webserver started at http://127.0.98.62:44311/ using document root <none> and password file <none>
I20260929 01:02:30.206934 5527 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:30.207028 5527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:30.207244 5527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:30.209160 5527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "d988801917c8460fa6579101016c0651"
format_stamp: "Formatted at 2026-09-29 01:02:30 on dist-test-slave-7zdl"
server_key: "e2b88af643e0099b733621844ea8f7fe"
server_key_iv: "ecd750651646badace046488a48de466"
server_key_version: "encryptionkey@0"
I20260929 01:02:30.209719 5527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "d988801917c8460fa6579101016c0651"
format_stamp: "Formatted at 2026-09-29 01:02:30 on dist-test-slave-7zdl"
server_key: "e2b88af643e0099b733621844ea8f7fe"
server_key_iv: "ecd750651646badace046488a48de466"
server_key_version: "encryptionkey@0"
I20260929 01:02:30.213483 5527 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:02:30.215829 5541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:30.217084 5527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260929 01:02:30.217180 5527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "d988801917c8460fa6579101016c0651"
format_stamp: "Formatted at 2026-09-29 01:02:30 on dist-test-slave-7zdl"
server_key: "e2b88af643e0099b733621844ea8f7fe"
server_key_iv: "ecd750651646badace046488a48de466"
server_key_version: "encryptionkey@0"
I20260929 01:02:30.217257 5527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:30.228391 5527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:30.228914 5527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:30.229066 5527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:30.235272 5527 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36629
I20260929 01:02:30.235278 5593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36629 every 8 connection(s)
I20260929 01:02:30.236145 5527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:02:30.239086 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 5527
I20260929 01:02:30.239194 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:02:30.239344 5594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:30.239483 392 external_mini_cluster.cc:1428] Setting key c892a0dc69ca23b1591c0bae6482ddd4
I20260929 01:02:30.246373 5594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d988801917c8460fa6579101016c0651: Bootstrap starting.
I20260929 01:02:30.248942 5594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d988801917c8460fa6579101016c0651: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:30.249473 5597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:30.240781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36048 (local address 127.0.98.62:36629)
0929 01:02:30.241423 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:30.241444 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:30.241489 (+ 45us) server_negotiation.cc:407] Connection header received
0929 01:02:30.242168 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:30.242191 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:30.242702 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:30.243105 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:30.243531 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:30.244944 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:30.245786 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:30.246457 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:30.246611 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:30.247456 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:30.247465 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:30.247481 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:02:30.247955 (+ 474us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:30.248031 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:02:30.248056 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:30.248144 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:30.248269 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 01:02:30.248522 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":107,"threads_started":1}
I20260929 01:02:30.249775 5594 log.cc:824] T 00000000000000000000000000000000 P d988801917c8460fa6579101016c0651: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:30.252220 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:02:30.252259 5594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d988801917c8460fa6579101016c0651: No bootstrap required, opened a new log
I20260929 01:02:30.255939 5594 raft_consensus.cc:348] T 00000000000000000000000000000000 P d988801917c8460fa6579101016c0651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d988801917c8460fa6579101016c0651" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36629 } }
I20260929 01:02:30.256150 5594 raft_consensus.cc:374] T 00000000000000000000000000000000 P d988801917c8460fa6579101016c0651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:30.256183 5594 raft_consensus.cc:729] T 00000000000000000000000000000000 P d988801917c8460fa6579101016c0651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d988801917c8460fa6579101016c0651, State: Initialized, Role: FOLLOWER
I20260929 01:02:30.256562 5594 consensus_queue.cc:260] T 00000000000000000000000000000000 P d988801917c8460fa6579101016c0651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d988801917c8460fa6579101016c0651" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36629 } }
I20260929 01:02:30.256677 5594 raft_consensus.cc:388] T 00000000000000000000000000000000 P d988801917c8460fa6579101016c0651 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:02:30.256722 5594 raft_consensus.cc:482] T 00000000000000000000000000000000 P d988801917c8460fa6579101016c0651 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:02:30.256783 5594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d988801917c8460fa6579101016c0651 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:30.257543 5594 raft_consensus.cc:504] T 00000000000000000000000000000000 P d988801917c8460fa6579101016c0651 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d988801917c8460fa6579101016c0651" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36629 } }
I20260929 01:02:30.257822 5594 leader_election.cc:302] T 00000000000000000000000000000000 P d988801917c8460fa6579101016c0651 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d988801917c8460fa6579101016c0651; no voters:
I20260929 01:02:30.258090 5594 leader_election.cc:288] T 00000000000000000000000000000000 P d988801917c8460fa6579101016c0651 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:02:30.258200 5599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d988801917c8460fa6579101016c0651 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:30.258415 5599 raft_consensus.cc:686] T 00000000000000000000000000000000 P d988801917c8460fa6579101016c0651 [term 1 LEADER]: Becoming Leader. State: Replica: d988801917c8460fa6579101016c0651, State: Running, Role: LEADER
I20260929 01:02:30.258697 5599 consensus_queue.cc:237] T 00000000000000000000000000000000 P d988801917c8460fa6579101016c0651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d988801917c8460fa6579101016c0651" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36629 } }
I20260929 01:02:30.259261 5594 sys_catalog.cc:564] T 00000000000000000000000000000000 P d988801917c8460fa6579101016c0651 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:02:30.259819 5600 sys_catalog.cc:455] T 00000000000000000000000000000000 P d988801917c8460fa6579101016c0651 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d988801917c8460fa6579101016c0651. Latest consensus state: current_term: 1 leader_uuid: "d988801917c8460fa6579101016c0651" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d988801917c8460fa6579101016c0651" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36629 } } }
I20260929 01:02:30.259951 5600 sys_catalog.cc:458] T 00000000000000000000000000000000 P d988801917c8460fa6579101016c0651 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:30.260248 5601 sys_catalog.cc:455] T 00000000000000000000000000000000 P d988801917c8460fa6579101016c0651 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d988801917c8460fa6579101016c0651" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d988801917c8460fa6579101016c0651" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36629 } } }
I20260929 01:02:30.260344 5601 sys_catalog.cc:458] T 00000000000000000000000000000000 P d988801917c8460fa6579101016c0651 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:30.260521 5605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:02:30.263397 5605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:02:30.268626 5605 catalog_manager.cc:1347] Generated new cluster ID: bc0c3c6715234c6889fc294c4cb7ede0
I20260929 01:02:30.268692 5605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:02:30.278168 5605 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:02:30.279233 5605 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:02:30.287505 5605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d988801917c8460fa6579101016c0651: Generated new TSK 0
I20260929 01:02:30.288204 5605 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 01:02:30.318845 5624 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 01:02:30.318987 5624 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 01:02:30.319566 5597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:30.301211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36060 (local address 127.0.98.62:36629)
0929 01:02:30.301348 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:30.301353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:30.301539 (+ 186us) server_negotiation.cc:407] Connection header received
0929 01:02:30.301734 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:30.301738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:30.301808 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:30.301929 (+ 121us) server_negotiation.cc:226] Negotiated authn=JWT
0929 01:02:30.302167 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:30.303027 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:30.303957 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:30.304313 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:30.304421 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:30.304718 (+ 297us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 01:02:30.319128 (+ 14410us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 01:02:30.319422 (+ 294us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36060: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 01:02:30.319641 5597 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36060: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 01:02:30.320869 5623 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:30.301082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36629 (local address 127.0.0.1:36060)
0929 01:02:30.301454 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:30.301466 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:30.301582 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:30.301935 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:30.301938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:30.301948 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0929 01:02:30.302068 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:30.302073 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:30.303152 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:30.303154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:30.303844 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:30.303852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:30.304409 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:30.304425 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:30.304496 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:30.304537 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 01:02:30.319344 (+ 14807us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 01:02:30.319564 (+ 220us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36629: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":117,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:36629: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43769/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 01:02:30.321723 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 5527
2026-09-29T01:02:30Z chronyd exiting
I20260929 01:02:30.359544 392 test_util.cc:175] -----------------------------------------------
I20260929 01:02:30.359629 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790643720886468-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (297 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 01:02:30.360689 392 mini_oidc.cc:152] Starting JWKS server
I20260929 01:02:30.361565 392 webserver.cc:460] Webserver started at https://127.0.0.1:44399/ using document root <none> and password file <none>
I20260929 01:02:30.361677 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 01:02:30.361943 392 webserver.cc:460] Webserver started at http://127.0.0.1:43503/ using document root <none> and password file <none>
2026-09-29T01:02:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:02:30Z Disabled control of system clock
I20260929 01:02:30.370559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34061
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34061
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44399/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643720886468-392-0/testchainca.cert with env {}
W20260929 01:02:30.481058 5630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:30.481305 5630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:30.481343 5630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:30.485280 5630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:02:30.485333 5630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:30.485349 5630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:30.485365 5630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:02:30.485380 5630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 01:02:30.485395 5630 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 01:02:30.485414 5630 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44399/jwks.json
I20260929 01:02:30.490607 5630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34061
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44399/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643720886468-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:30.491590 5630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:30.492492 5630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:30.500540 5635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:30.500578 5636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:30.500698 5630 server_base.cc:1034] running on GCE node
W20260929 01:02:30.500633 5638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:30.501418 5630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:30.502475 5630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:30.503645 5630 hybrid_clock.cc:648] HybridClock initialized: now 1790643750503618 us; error 40 us; skew 500 ppm
I20260929 01:02:30.505505 5630 webserver.cc:460] Webserver started at http://127.0.98.62:39409/ using document root <none> and password file <none>
I20260929 01:02:30.506058 5630 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:30.506143 5630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:30.506352 5630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:30.508332 5630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "6566530bfe4c49bb892d2a705ef8703f"
format_stamp: "Formatted at 2026-09-29 01:02:30 on dist-test-slave-7zdl"
server_key: "1c00c95d9976bf4b77666a92972edbe7"
server_key_iv: "d2fc3d51c141f18727f3896b8a42f165"
server_key_version: "encryptionkey@0"
I20260929 01:02:30.508869 5630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "6566530bfe4c49bb892d2a705ef8703f"
format_stamp: "Formatted at 2026-09-29 01:02:30 on dist-test-slave-7zdl"
server_key: "1c00c95d9976bf4b77666a92972edbe7"
server_key_iv: "d2fc3d51c141f18727f3896b8a42f165"
server_key_version: "encryptionkey@0"
I20260929 01:02:30.512354 5630 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260929 01:02:30.514634 5644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:30.515861 5630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:30.515995 5630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "6566530bfe4c49bb892d2a705ef8703f"
format_stamp: "Formatted at 2026-09-29 01:02:30 on dist-test-slave-7zdl"
server_key: "1c00c95d9976bf4b77666a92972edbe7"
server_key_iv: "d2fc3d51c141f18727f3896b8a42f165"
server_key_version: "encryptionkey@0"
I20260929 01:02:30.516099 5630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:30.533965 5630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:30.534673 5630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:30.534927 5630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:30.541720 5630 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34061
I20260929 01:02:30.541749 5696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34061 every 8 connection(s)
I20260929 01:02:30.542635 5630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:02:30.545691 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:30.546064 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 5630
I20260929 01:02:30.546193 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:02:30.546475 392 external_mini_cluster.cc:1428] Setting key 362ae377b35c95615d4c40b8bd04f1cd
I20260929 01:02:30.551513 5697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6566530bfe4c49bb892d2a705ef8703f: Bootstrap starting.
I20260929 01:02:30.553606 5697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6566530bfe4c49bb892d2a705ef8703f: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:30.554322 5697 log.cc:824] T 00000000000000000000000000000000 P 6566530bfe4c49bb892d2a705ef8703f: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:30.555303 5700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:30.547808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51832 (local address 127.0.98.62:34061)
0929 01:02:30.548419 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:30.548437 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:30.548472 (+ 35us) server_negotiation.cc:407] Connection header received
0929 01:02:30.549097 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:30.549113 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:30.549493 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:30.549818 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:30.550216 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:30.551215 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:30.552109 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:30.552575 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:30.552744 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:30.553558 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:30.553563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:30.553575 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:02:30.554005 (+ 430us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:30.554059 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:02:30.554078 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:30.554178 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:30.554291 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 01:02:30.554536 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":128,"threads_started":1}
I20260929 01:02:30.556438 5697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6566530bfe4c49bb892d2a705ef8703f: No bootstrap required, opened a new log
I20260929 01:02:30.557940 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:02:30.560222 5697 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6566530bfe4c49bb892d2a705ef8703f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6566530bfe4c49bb892d2a705ef8703f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34061 } }
I20260929 01:02:30.560451 5697 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6566530bfe4c49bb892d2a705ef8703f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:30.560532 5697 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6566530bfe4c49bb892d2a705ef8703f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6566530bfe4c49bb892d2a705ef8703f, State: Initialized, Role: FOLLOWER
I20260929 01:02:30.560957 5697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6566530bfe4c49bb892d2a705ef8703f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6566530bfe4c49bb892d2a705ef8703f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34061 } }
I20260929 01:02:30.561107 5697 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6566530bfe4c49bb892d2a705ef8703f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:02:30.561187 5697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6566530bfe4c49bb892d2a705ef8703f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:02:30.561282 5697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6566530bfe4c49bb892d2a705ef8703f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:30.562121 5697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6566530bfe4c49bb892d2a705ef8703f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6566530bfe4c49bb892d2a705ef8703f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34061 } }
I20260929 01:02:30.562510 5697 leader_election.cc:302] T 00000000000000000000000000000000 P 6566530bfe4c49bb892d2a705ef8703f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6566530bfe4c49bb892d2a705ef8703f; no voters:
I20260929 01:02:30.562842 5697 leader_election.cc:288] T 00000000000000000000000000000000 P 6566530bfe4c49bb892d2a705ef8703f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:02:30.562973 5702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6566530bfe4c49bb892d2a705ef8703f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:30.563181 5702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6566530bfe4c49bb892d2a705ef8703f [term 1 LEADER]: Becoming Leader. State: Replica: 6566530bfe4c49bb892d2a705ef8703f, State: Running, Role: LEADER
I20260929 01:02:30.563467 5702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6566530bfe4c49bb892d2a705ef8703f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6566530bfe4c49bb892d2a705ef8703f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34061 } }
I20260929 01:02:30.564080 5697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6566530bfe4c49bb892d2a705ef8703f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:02:30.564991 5704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6566530bfe4c49bb892d2a705ef8703f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6566530bfe4c49bb892d2a705ef8703f. Latest consensus state: current_term: 1 leader_uuid: "6566530bfe4c49bb892d2a705ef8703f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6566530bfe4c49bb892d2a705ef8703f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34061 } } }
I20260929 01:02:30.565126 5704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6566530bfe4c49bb892d2a705ef8703f [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:30.565428 5703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6566530bfe4c49bb892d2a705ef8703f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6566530bfe4c49bb892d2a705ef8703f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6566530bfe4c49bb892d2a705ef8703f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34061 } } }
I20260929 01:02:30.565522 5711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:02:30.565598 5703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6566530bfe4c49bb892d2a705ef8703f [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:30.568492 5711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:02:30.573820 5711 catalog_manager.cc:1347] Generated new cluster ID: 266b907305384573b385afa288385b50
I20260929 01:02:30.573889 5711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:02:30.585712 5711 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:02:30.586993 5711 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:02:30.593909 5711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6566530bfe4c49bb892d2a705ef8703f: Generated new TSK 0
I20260929 01:02:30.594710 5711 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 01:02:30.625102 5727 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 01:02:30.625196 5727 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 01:02:30.625801 5700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:30.602686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51838 (local address 127.0.98.62:34061)
0929 01:02:30.602859 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:30.602864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:30.602905 (+ 41us) server_negotiation.cc:407] Connection header received
0929 01:02:30.603067 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:30.603069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:30.603119 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:30.603215 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0929 01:02:30.603502 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:30.604058 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:30.605300 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:30.605503 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:30.605571 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:30.605861 (+ 290us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 01:02:30.625448 (+ 19587us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44399/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 01:02:30.625686 (+ 238us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51838: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44399/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 01:02:30.625883 5700 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51838: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44399/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 01:02:30.626004 5726 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:30.602454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34061 (local address 127.0.0.1:51838)
0929 01:02:30.602827 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:30.602842 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:30.602957 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:30.603234 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:30.603238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:30.603246 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 01:02:30.603362 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:30.603368 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:30.604172 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:30.604175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:30.605182 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:30.605191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:30.605585 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:30.605604 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:30.605690 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:30.605716 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 01:02:30.625649 (+ 19933us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44399/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 01:02:30.625829 (+ 180us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34061: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44399/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":144,"threads_started":1}
I20260929 01:02:30.626767 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 5630
2026-09-29T01:02:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (305 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 01:02:30.665714 392 mini_oidc.cc:152] Starting JWKS server
I20260929 01:02:30.666617 392 webserver.cc:460] Webserver started at https://127.0.0.1:40913/ using document root <none> and password file <none>
I20260929 01:02:30.666733 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 01:02:30.667092 392 webserver.cc:460] Webserver started at http://127.0.0.1:46587/ using document root <none> and password file <none>
2026-09-29T01:02:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:02:30Z Disabled control of system clock
I20260929 01:02:30.676002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42545
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42545
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40913/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 01:02:30.783872 5733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:30.784138 5733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:30.784176 5733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:30.788164 5733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:02:30.788223 5733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:30.788239 5733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:30.788255 5733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:02:30.788270 5733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 01:02:30.788292 5733 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 01:02:30.788306 5733 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40913/jwks.json
I20260929 01:02:30.792995 5733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42545
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40913/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:30.794021 5733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:30.795351 5733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:30.803653 5738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:30.803665 5739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:30.803754 5733 server_base.cc:1034] running on GCE node
W20260929 01:02:30.803665 5741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:30.804436 5733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:30.805526 5733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:30.806701 5733 hybrid_clock.cc:648] HybridClock initialized: now 1790643750806686 us; error 27 us; skew 500 ppm
I20260929 01:02:30.808528 5733 webserver.cc:460] Webserver started at http://127.0.98.62:33797/ using document root <none> and password file <none>
I20260929 01:02:30.809090 5733 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:30.809178 5733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:30.809391 5733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:30.811534 5733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "c163fa9b84484c52b1c88ab3ec7d6835"
format_stamp: "Formatted at 2026-09-29 01:02:30 on dist-test-slave-7zdl"
server_key: "d8ee6adeb1b53bea8accfe31716219a8"
server_key_iv: "bac9cd63b76aa310b524f55d68871c85"
server_key_version: "encryptionkey@0"
I20260929 01:02:30.812081 5733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "c163fa9b84484c52b1c88ab3ec7d6835"
format_stamp: "Formatted at 2026-09-29 01:02:30 on dist-test-slave-7zdl"
server_key: "d8ee6adeb1b53bea8accfe31716219a8"
server_key_iv: "bac9cd63b76aa310b524f55d68871c85"
server_key_version: "encryptionkey@0"
I20260929 01:02:30.815653 5733 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:02:30.817919 5747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:30.819123 5733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:02:30.819242 5733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "c163fa9b84484c52b1c88ab3ec7d6835"
format_stamp: "Formatted at 2026-09-29 01:02:30 on dist-test-slave-7zdl"
server_key: "d8ee6adeb1b53bea8accfe31716219a8"
server_key_iv: "bac9cd63b76aa310b524f55d68871c85"
server_key_version: "encryptionkey@0"
I20260929 01:02:30.819342 5733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:30.849964 5733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:30.850682 5733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:30.850924 5733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:30.857532 5733 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42545
I20260929 01:02:30.857543 5799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42545 every 8 connection(s)
I20260929 01:02:30.858428 5733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:02:30.861482 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 5733
I20260929 01:02:30.861554 5800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:30.861609 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:02:30.861865 392 external_mini_cluster.cc:1428] Setting key f2c440f49b9f11c0a0e6d41b5b483382
I20260929 01:02:30.867799 5800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c163fa9b84484c52b1c88ab3ec7d6835: Bootstrap starting.
I20260929 01:02:30.869979 5800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c163fa9b84484c52b1c88ab3ec7d6835: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:30.870762 5800 log.cc:824] T 00000000000000000000000000000000 P c163fa9b84484c52b1c88ab3ec7d6835: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:30.871098 5803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:30.863199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57044 (local address 127.0.98.62:42545)
0929 01:02:30.863797 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:30.863810 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:30.863844 (+ 34us) server_negotiation.cc:407] Connection header received
0929 01:02:30.864670 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:30.864689 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:30.865056 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:30.865429 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:30.865840 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:30.866916 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:30.867889 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:30.868380 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:30.868549 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:30.869398 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:30.869403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:30.869414 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:02:30.869787 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:30.869840 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:02:30.869859 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:30.869959 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:30.870075 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 01:02:30.870321 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":134,"threads_started":1}
I20260929 01:02:30.872613 5800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c163fa9b84484c52b1c88ab3ec7d6835: No bootstrap required, opened a new log
I20260929 01:02:30.873728 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:02:30.876508 5800 raft_consensus.cc:348] T 00000000000000000000000000000000 P c163fa9b84484c52b1c88ab3ec7d6835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c163fa9b84484c52b1c88ab3ec7d6835" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42545 } }
I20260929 01:02:30.876827 5800 raft_consensus.cc:374] T 00000000000000000000000000000000 P c163fa9b84484c52b1c88ab3ec7d6835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:30.876890 5800 raft_consensus.cc:729] T 00000000000000000000000000000000 P c163fa9b84484c52b1c88ab3ec7d6835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c163fa9b84484c52b1c88ab3ec7d6835, State: Initialized, Role: FOLLOWER
I20260929 01:02:30.877307 5800 consensus_queue.cc:260] T 00000000000000000000000000000000 P c163fa9b84484c52b1c88ab3ec7d6835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c163fa9b84484c52b1c88ab3ec7d6835" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42545 } }
I20260929 01:02:30.877449 5800 raft_consensus.cc:388] T 00000000000000000000000000000000 P c163fa9b84484c52b1c88ab3ec7d6835 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:02:30.877530 5800 raft_consensus.cc:482] T 00000000000000000000000000000000 P c163fa9b84484c52b1c88ab3ec7d6835 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:02:30.877642 5800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c163fa9b84484c52b1c88ab3ec7d6835 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:30.878520 5800 raft_consensus.cc:504] T 00000000000000000000000000000000 P c163fa9b84484c52b1c88ab3ec7d6835 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c163fa9b84484c52b1c88ab3ec7d6835" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42545 } }
I20260929 01:02:30.878912 5800 leader_election.cc:302] T 00000000000000000000000000000000 P c163fa9b84484c52b1c88ab3ec7d6835 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c163fa9b84484c52b1c88ab3ec7d6835; no voters:
I20260929 01:02:30.879230 5800 leader_election.cc:288] T 00000000000000000000000000000000 P c163fa9b84484c52b1c88ab3ec7d6835 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:02:30.879339 5805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c163fa9b84484c52b1c88ab3ec7d6835 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:30.879590 5805 raft_consensus.cc:686] T 00000000000000000000000000000000 P c163fa9b84484c52b1c88ab3ec7d6835 [term 1 LEADER]: Becoming Leader. State: Replica: c163fa9b84484c52b1c88ab3ec7d6835, State: Running, Role: LEADER
I20260929 01:02:30.879880 5805 consensus_queue.cc:237] T 00000000000000000000000000000000 P c163fa9b84484c52b1c88ab3ec7d6835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c163fa9b84484c52b1c88ab3ec7d6835" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42545 } }
I20260929 01:02:30.880468 5800 sys_catalog.cc:564] T 00000000000000000000000000000000 P c163fa9b84484c52b1c88ab3ec7d6835 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:02:30.881587 5806 sys_catalog.cc:455] T 00000000000000000000000000000000 P c163fa9b84484c52b1c88ab3ec7d6835 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c163fa9b84484c52b1c88ab3ec7d6835" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c163fa9b84484c52b1c88ab3ec7d6835" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42545 } } }
I20260929 01:02:30.881709 5806 sys_catalog.cc:458] T 00000000000000000000000000000000 P c163fa9b84484c52b1c88ab3ec7d6835 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:30.881980 5807 sys_catalog.cc:455] T 00000000000000000000000000000000 P c163fa9b84484c52b1c88ab3ec7d6835 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c163fa9b84484c52b1c88ab3ec7d6835. Latest consensus state: current_term: 1 leader_uuid: "c163fa9b84484c52b1c88ab3ec7d6835" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c163fa9b84484c52b1c88ab3ec7d6835" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42545 } } }
I20260929 01:02:30.882069 5807 sys_catalog.cc:458] T 00000000000000000000000000000000 P c163fa9b84484c52b1c88ab3ec7d6835 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:30.882195 5814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:02:30.885406 5814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:02:30.898010 5814 catalog_manager.cc:1347] Generated new cluster ID: 86556fc8333c46c2892ef56054de4969
I20260929 01:02:30.898105 5814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:02:30.919816 5814 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:02:30.921072 5814 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:02:30.930663 5814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c163fa9b84484c52b1c88ab3ec7d6835: Generated new TSK 0
I20260929 01:02:30.931517 5814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:02:30.950613 5803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:30.945248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57052 (local address 127.0.98.62:42545)
0929 01:02:30.945373 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:30.945378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:30.945570 (+ 192us) server_negotiation.cc:407] Connection header received
0929 01:02:30.945717 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:30.945720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:30.945764 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:30.945851 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0929 01:02:30.946079 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:30.946608 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:30.947838 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:30.948044 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:30.948115 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:30.948415 (+ 300us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 01:02:30.950239 (+ 1824us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40913/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 01:02:30.950473 (+ 234us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57052: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40913/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 01:02:30.950676 5803 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57052: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40913/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 01:02:30.952556 5829 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:30.945195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42545 (local address 127.0.0.1:57052)
0929 01:02:30.945480 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:30.945494 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:30.945609 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:30.945868 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:30.945871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:30.945878 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0929 01:02:30.945984 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:30.945990 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:30.946706 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:30.946710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:30.947728 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:30.947739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:30.948127 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:30.948151 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:30.948239 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:30.948265 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 01:02:30.950406 (+ 2141us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40913/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 01:02:30.952349 (+ 1943us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42545: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40913/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":1669,"thread_start_us":109,"threads_started":1}
I20260929 01:02:30.953096 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 5733
E20260929 01:02:30.961494 5830 webserver.cc:533] Webserver: sslize failed:
E20260929 01:02:30.961553 5830 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T01:02:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (326 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T01:02:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:02:30Z Disabled control of system clock
I20260929 01:02:31.000185 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43987
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43987
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643720886468-392-0/testchainca.cert with env {}
W20260929 01:02:31.112031 5834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:31.112290 5834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:31.112331 5834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:31.116199 5834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:02:31.116252 5834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:31.116271 5834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:31.116289 5834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:02:31.116307 5834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 01:02:31.116333 5834 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 01:02:31.116350 5834 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 01:02:31.121387 5834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43987
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643720886468-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:31.122269 5834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:31.123224 5834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:31.131726 5842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:31.131783 5839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:31.131976 5834 server_base.cc:1034] running on GCE node
W20260929 01:02:31.132174 5840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:31.132701 5834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:31.133742 5834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:31.134879 5834 hybrid_clock.cc:648] HybridClock initialized: now 1790643751134864 us; error 28 us; skew 500 ppm
I20260929 01:02:31.136906 5834 webserver.cc:460] Webserver started at http://127.0.98.62:38367/ using document root <none> and password file <none>
I20260929 01:02:31.137477 5834 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:31.137568 5834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:31.137817 5834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:31.139889 5834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "933d4046f58348edaf3d789f25bd103b"
format_stamp: "Formatted at 2026-09-29 01:02:31 on dist-test-slave-7zdl"
server_key: "e3675a3da1c7a7a6851510dce9ad3751"
server_key_iv: "f8b0c27c4b43da9eb17e253f1bbf0afe"
server_key_version: "encryptionkey@0"
I20260929 01:02:31.140450 5834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "933d4046f58348edaf3d789f25bd103b"
format_stamp: "Formatted at 2026-09-29 01:02:31 on dist-test-slave-7zdl"
server_key: "e3675a3da1c7a7a6851510dce9ad3751"
server_key_iv: "f8b0c27c4b43da9eb17e253f1bbf0afe"
server_key_version: "encryptionkey@0"
I20260929 01:02:31.143980 5834 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:02:31.146762 5848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:31.148022 5834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:02:31.148185 5834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "933d4046f58348edaf3d789f25bd103b"
format_stamp: "Formatted at 2026-09-29 01:02:31 on dist-test-slave-7zdl"
server_key: "e3675a3da1c7a7a6851510dce9ad3751"
server_key_iv: "f8b0c27c4b43da9eb17e253f1bbf0afe"
server_key_version: "encryptionkey@0"
I20260929 01:02:31.148298 5834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:31.164745 5834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:31.165361 5834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:31.165550 5834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:31.172339 5834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43987
I20260929 01:02:31.172364 5900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43987 every 8 connection(s)
I20260929 01:02:31.173272 5834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:02:31.175282 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 5834
I20260929 01:02:31.175405 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:02:31.175712 392 external_mini_cluster.cc:1428] Setting key c94d70178bed8d8caf3f3af6c3871d7b
I20260929 01:02:31.176726 5901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:31.182122 5901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 933d4046f58348edaf3d789f25bd103b: Bootstrap starting.
I20260929 01:02:31.184442 5901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 933d4046f58348edaf3d789f25bd103b: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:31.184484 5904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:31.177012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44198 (local address 127.0.98.62:43987)
0929 01:02:31.177742 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:31.177758 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:31.177794 (+ 36us) server_negotiation.cc:407] Connection header received
0929 01:02:31.178441 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:31.178458 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:31.178881 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:31.179207 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:31.179602 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:31.180532 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:31.181381 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:31.181839 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:31.182018 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:31.182878 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:31.182882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:31.182899 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:02:31.183259 (+ 360us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:31.183310 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:02:31.183329 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:31.183435 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:31.183580 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 01:02:31.183813 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":129,"threads_started":1}
I20260929 01:02:31.185329 5901 log.cc:824] T 00000000000000000000000000000000 P 933d4046f58348edaf3d789f25bd103b: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:31.187217 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:02:31.187427 5901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 933d4046f58348edaf3d789f25bd103b: No bootstrap required, opened a new log
I20260929 01:02:31.190915 5901 raft_consensus.cc:348] T 00000000000000000000000000000000 P 933d4046f58348edaf3d789f25bd103b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933d4046f58348edaf3d789f25bd103b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43987 } }
I20260929 01:02:31.191150 5901 raft_consensus.cc:374] T 00000000000000000000000000000000 P 933d4046f58348edaf3d789f25bd103b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:31.191249 5901 raft_consensus.cc:729] T 00000000000000000000000000000000 P 933d4046f58348edaf3d789f25bd103b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 933d4046f58348edaf3d789f25bd103b, State: Initialized, Role: FOLLOWER
I20260929 01:02:31.191720 5901 consensus_queue.cc:260] T 00000000000000000000000000000000 P 933d4046f58348edaf3d789f25bd103b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933d4046f58348edaf3d789f25bd103b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43987 } }
I20260929 01:02:31.191891 5901 raft_consensus.cc:388] T 00000000000000000000000000000000 P 933d4046f58348edaf3d789f25bd103b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:02:31.191958 5901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 933d4046f58348edaf3d789f25bd103b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:02:31.192059 5901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 933d4046f58348edaf3d789f25bd103b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:31.192905 5901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 933d4046f58348edaf3d789f25bd103b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933d4046f58348edaf3d789f25bd103b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43987 } }
I20260929 01:02:31.193228 5901 leader_election.cc:302] T 00000000000000000000000000000000 P 933d4046f58348edaf3d789f25bd103b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 933d4046f58348edaf3d789f25bd103b; no voters:
I20260929 01:02:31.193514 5901 leader_election.cc:288] T 00000000000000000000000000000000 P 933d4046f58348edaf3d789f25bd103b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:02:31.193600 5906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 933d4046f58348edaf3d789f25bd103b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:31.193903 5906 raft_consensus.cc:686] T 00000000000000000000000000000000 P 933d4046f58348edaf3d789f25bd103b [term 1 LEADER]: Becoming Leader. State: Replica: 933d4046f58348edaf3d789f25bd103b, State: Running, Role: LEADER
I20260929 01:02:31.194186 5906 consensus_queue.cc:237] T 00000000000000000000000000000000 P 933d4046f58348edaf3d789f25bd103b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933d4046f58348edaf3d789f25bd103b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43987 } }
I20260929 01:02:31.194703 5901 sys_catalog.cc:564] T 00000000000000000000000000000000 P 933d4046f58348edaf3d789f25bd103b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:02:31.195879 5908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 933d4046f58348edaf3d789f25bd103b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 933d4046f58348edaf3d789f25bd103b. Latest consensus state: current_term: 1 leader_uuid: "933d4046f58348edaf3d789f25bd103b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933d4046f58348edaf3d789f25bd103b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43987 } } }
I20260929 01:02:31.195879 5907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 933d4046f58348edaf3d789f25bd103b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "933d4046f58348edaf3d789f25bd103b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "933d4046f58348edaf3d789f25bd103b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43987 } } }
I20260929 01:02:31.196030 5908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 933d4046f58348edaf3d789f25bd103b [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:31.196030 5907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 933d4046f58348edaf3d789f25bd103b [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:31.196903 5915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:02:31.199533 5915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:02:31.204815 5915 catalog_manager.cc:1347] Generated new cluster ID: bedda2e87f544799b8c9a6fc08c5e933
I20260929 01:02:31.204886 5915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:02:31.211558 5915 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:02:31.212356 5915 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:02:31.218164 5915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 933d4046f58348edaf3d789f25bd103b: Generated new TSK 0
I20260929 01:02:31.218726 5915 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:02:31.274767 5904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:31.226213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44206 (local address 127.0.98.62:43987)
0929 01:02:31.226339 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:31.226344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:31.226517 (+ 173us) server_negotiation.cc:407] Connection header received
0929 01:02:31.226660 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:31.226662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:31.226706 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:31.226781 (+ 75us) server_negotiation.cc:226] Negotiated authn=JWT
0929 01:02:31.227050 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:31.227539 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:31.228750 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:31.228948 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:31.229026 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:31.229323 (+ 297us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 01:02:31.274378 (+ 45055us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 01:02:31.274650 (+ 272us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44206: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 01:02:31.274921 5904 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44206: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 01:02:31.275334 5930 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:31.226150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43987 (local address 127.0.0.1:44206)
0929 01:02:31.226430 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:31.226444 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:31.226550 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:31.226825 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:31.226828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:31.226836 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 01:02:31.226947 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:31.226954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:31.227654 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:31.227658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:31.228643 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:31.228653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:31.229045 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:31.229067 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:31.229154 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:31.229181 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 01:02:31.274623 (+ 45442us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 01:02:31.274857 (+ 234us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43987: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":98,"threads_started":1}
I20260929 01:02:31.276253 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 5834
2026-09-29T01:02:31Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (302 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (118 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (118 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:02:31 dist-test-slave-7zdl krb5kdc[5942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:02:31 dist-test-slave-7zdl krb5kdc[5942](info): set up 2 sockets
Sep 29 01:02:31 dist-test-slave-7zdl krb5kdc[5942](info): commencing operation
krb5kdc: starting...
W20260929 01:02:32.551188 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:02:32 dist-test-slave-7zdl krb5kdc[5942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:02:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:02:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:02:32.669104 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43851
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43851
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:32.774358 5955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:32.774667 5955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:32.774737 5955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:32.778640 5955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:02:32.778728 5955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:32.778764 5955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:32.778818 5955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:02:32.778874 5955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:02:32.783996 5955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:32.785003 5955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:32.785982 5955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:02:32.794337 5955 server_base.cc:1034] running on GCE node
W20260929 01:02:32.794337 5960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:32.794374 5961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:32.794492 5963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:32.795092 5955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:32.796166 5955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:32.797318 5955 hybrid_clock.cc:648] HybridClock initialized: now 1790643752797296 us; error 37 us; skew 500 ppm
Sep 29 01:02:32 dist-test-slave-7zdl krb5kdc[5942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:32.800098 5955 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:02:32.800879 5955 webserver.cc:460] Webserver started at http://127.0.98.62:41145/ using document root <none> and password file <none>
I20260929 01:02:32.801404 5955 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:32.801487 5955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:32.801697 5955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:32.803681 5955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "3b52cd0b62c9444da283d6cca6569a31"
format_stamp: "Formatted at 2026-09-29 01:02:32 on dist-test-slave-7zdl"
server_key: "18a3b87f15f5699c90c38d319d49cb73"
server_key_iv: "512d3e1427a8c6acd9916f119dbe0cdd"
server_key_version: "encryptionkey@0"
I20260929 01:02:32.804255 5955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "3b52cd0b62c9444da283d6cca6569a31"
format_stamp: "Formatted at 2026-09-29 01:02:32 on dist-test-slave-7zdl"
server_key: "18a3b87f15f5699c90c38d319d49cb73"
server_key_iv: "512d3e1427a8c6acd9916f119dbe0cdd"
server_key_version: "encryptionkey@0"
I20260929 01:02:32.807803 5955 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:02:32.810058 5970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:32.811331 5955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:02:32.811472 5955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "3b52cd0b62c9444da283d6cca6569a31"
format_stamp: "Formatted at 2026-09-29 01:02:32 on dist-test-slave-7zdl"
server_key: "18a3b87f15f5699c90c38d319d49cb73"
server_key_iv: "512d3e1427a8c6acd9916f119dbe0cdd"
server_key_version: "encryptionkey@0"
I20260929 01:02:32.811580 5955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:32.829648 5955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:32.832460 5955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:32.832659 5955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:32.839399 5955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43851
I20260929 01:02:32.839692 6022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43851 every 8 connection(s)
I20260929 01:02:32.840390 5955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:02:32.843142 6023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:32.844523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 5955
I20260929 01:02:32.844657 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:02:32.844938 392 external_mini_cluster.cc:1428] Setting key 328992553fdf43b6bae9a71bb763e159
I20260929 01:02:32.849653 6023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b52cd0b62c9444da283d6cca6569a31: Bootstrap starting.
I20260929 01:02:32.852237 6023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b52cd0b62c9444da283d6cca6569a31: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:32.853091 6023 log.cc:824] T 00000000000000000000000000000000 P 3b52cd0b62c9444da283d6cca6569a31: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:02:32 dist-test-slave-7zdl krb5kdc[5942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:32.855011 6023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b52cd0b62c9444da283d6cca6569a31: No bootstrap required, opened a new log
I20260929 01:02:32.858428 6023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b52cd0b62c9444da283d6cca6569a31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b52cd0b62c9444da283d6cca6569a31" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43851 } }
I20260929 01:02:32.858670 6023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b52cd0b62c9444da283d6cca6569a31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:32.858757 6023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b52cd0b62c9444da283d6cca6569a31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b52cd0b62c9444da283d6cca6569a31, State: Initialized, Role: FOLLOWER
I20260929 01:02:32.859239 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:32.846320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51822 (local address 127.0.98.62:43851)
0929 01:02:32.846723 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:32.846741 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:32.846785 (+ 44us) server_negotiation.cc:407] Connection header received
0929 01:02:32.847624 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:32.847649 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:32.848064 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:32.848431 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:32.849063 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:32.850523 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:32.851403 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:32.852077 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:32.852256 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:32.854747 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:32.854752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:32.854765 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:32.854790 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:32.856449 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:32.856894 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:32.856899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:32.856903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:32.857011 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:32.857289 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:32.857293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:32.857295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:32.857663 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:32.857884 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:32.858134 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 01:02:32.858387 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":98,"threads_started":1}
I20260929 01:02:32.859231 6023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b52cd0b62c9444da283d6cca6569a31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b52cd0b62c9444da283d6cca6569a31" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43851 } }
I20260929 01:02:32.859403 6023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b52cd0b62c9444da283d6cca6569a31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:02:32.859470 6023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b52cd0b62c9444da283d6cca6569a31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:02:32.859611 6023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b52cd0b62c9444da283d6cca6569a31 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:32.860519 6023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b52cd0b62c9444da283d6cca6569a31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b52cd0b62c9444da283d6cca6569a31" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43851 } }
I20260929 01:02:32.860847 6023 leader_election.cc:302] T 00000000000000000000000000000000 P 3b52cd0b62c9444da283d6cca6569a31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b52cd0b62c9444da283d6cca6569a31; no voters:
I20260929 01:02:32.861136 6023 leader_election.cc:288] T 00000000000000000000000000000000 P 3b52cd0b62c9444da283d6cca6569a31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:02:32.861269 6028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b52cd0b62c9444da283d6cca6569a31 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:32.861481 6028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b52cd0b62c9444da283d6cca6569a31 [term 1 LEADER]: Becoming Leader. State: Replica: 3b52cd0b62c9444da283d6cca6569a31, State: Running, Role: LEADER
I20260929 01:02:32.861743 6028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b52cd0b62c9444da283d6cca6569a31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b52cd0b62c9444da283d6cca6569a31" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43851 } }
I20260929 01:02:32.862237 6023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b52cd0b62c9444da283d6cca6569a31 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:02:32.862857 6028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b52cd0b62c9444da283d6cca6569a31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b52cd0b62c9444da283d6cca6569a31. Latest consensus state: current_term: 1 leader_uuid: "3b52cd0b62c9444da283d6cca6569a31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b52cd0b62c9444da283d6cca6569a31" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43851 } } }
I20260929 01:02:32.862973 6028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b52cd0b62c9444da283d6cca6569a31 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:32.863334 6034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:02:32.863343 6029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b52cd0b62c9444da283d6cca6569a31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b52cd0b62c9444da283d6cca6569a31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b52cd0b62c9444da283d6cca6569a31" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43851 } } }
I20260929 01:02:32.863533 6029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b52cd0b62c9444da283d6cca6569a31 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:32.866262 6034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:02:32.871155 6034 catalog_manager.cc:1347] Generated new cluster ID: 04272f4307614c539ba93009705ed454
I20260929 01:02:32.871213 6034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:02:32.882097 6034 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:02:32.883074 6034 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:02:32.887230 6034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b52cd0b62c9444da283d6cca6569a31: Generated new TSK 0
I20260929 01:02:32.887852 6034 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 01:02:32.921690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43851
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:33.028345 6049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:33.028632 6049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:33.028704 6049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:33.032733 6049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:33.032810 6049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:33.032892 6049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:02:33.037890 6049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:33.038883 6049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:33.039839 6049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:33.049153 6055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:33.049242 6054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:33.049244 6049 server_base.cc:1034] running on GCE node
W20260929 01:02:33.049350 6057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:33.049702 6049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:33.050321 6049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:33.051479 6049 hybrid_clock.cc:648] HybridClock initialized: now 1790643753051466 us; error 23 us; skew 500 ppm
Sep 29 01:02:33 dist-test-slave-7zdl krb5kdc[5942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:33.058314 6049 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:02:33.059154 6049 webserver.cc:460] Webserver started at http://127.0.98.1:37441/ using document root <none> and password file <none>
I20260929 01:02:33.059650 6049 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:33.059705 6049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:33.059857 6049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:33.061704 6049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "53407a78f69746f5bf30346ba9143754"
format_stamp: "Formatted at 2026-09-29 01:02:33 on dist-test-slave-7zdl"
server_key: "d99a0d5d6f3b488b7096769450d3d397"
server_key_iv: "cda3e233363c7c31cdd3da9fcd009ab2"
server_key_version: "encryptionkey@0"
I20260929 01:02:33.062218 6049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "53407a78f69746f5bf30346ba9143754"
format_stamp: "Formatted at 2026-09-29 01:02:33 on dist-test-slave-7zdl"
server_key: "d99a0d5d6f3b488b7096769450d3d397"
server_key_iv: "cda3e233363c7c31cdd3da9fcd009ab2"
server_key_version: "encryptionkey@0"
I20260929 01:02:33.065852 6049 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 01:02:33.068156 6064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:33.069267 6049 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:02:33.069382 6049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "53407a78f69746f5bf30346ba9143754"
format_stamp: "Formatted at 2026-09-29 01:02:33 on dist-test-slave-7zdl"
server_key: "d99a0d5d6f3b488b7096769450d3d397"
server_key_iv: "cda3e233363c7c31cdd3da9fcd009ab2"
server_key_version: "encryptionkey@0"
I20260929 01:02:33.069478 6049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:33.095498 6049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:33.098392 6049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:33.098606 6049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:33.099217 6049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:33.100011 6049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:33.100056 6049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:33.100124 6049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:33.100154 6049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:33.109357 6049 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38897
I20260929 01:02:33.109385 6177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38897 every 8 connection(s)
I20260929 01:02:33.110265 6049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:02:33.117877 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 6049
I20260929 01:02:33.118034 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:02:33.118325 392 external_mini_cluster.cc:1428] Setting key f3b02777451162a15abc5cbe7af9f9bd
Sep 29 01:02:33 dist-test-slave-7zdl krb5kdc[5942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:33.124161 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:33.112021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51431 (local address 127.0.98.62:43851)
0929 01:02:33.112153 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:33.112157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:33.112994 (+ 837us) server_negotiation.cc:407] Connection header received
0929 01:02:33.114357 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:33.114361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:33.114430 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:33.114513 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:33.115776 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:33.116302 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:33.117450 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:33.117783 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:33.117896 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:33.120666 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:33.120670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:33.120672 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:33.120700 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:33.122151 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:33.122651 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:33.122656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:33.122658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:33.122738 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:33.123095 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:33.123100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:33.123102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:33.123277 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:33.123377 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:33.123850 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 01:02:33.123991 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:02:33.124898 6180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:33.112416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43851 (local address 127.0.98.1:51431)
0929 01:02:33.112845 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:33.112894 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:33.114149 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:33.114677 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:33.114684 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:33.115107 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:33.115640 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:33.115655 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:33.116446 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:33.116449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:33.117323 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:33.117332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:33.117945 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:33.117948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:33.118185 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:33.118911 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:33.118941 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:33.120504 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:33.122281 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:33.122286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:33.122301 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:33.122551 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:33.122856 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:33.122860 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:33.122862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:33.123002 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:33.123386 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:33.123393 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:33.123623 (+ 230us) client_negotiation.cc:769] Sending connection context
0929 01:02:33.123797 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 01:02:33.124125 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":112,"threads_started":1}
I20260929 01:02:33.126047 6178 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43851
I20260929 01:02:33.126302 6178 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:33.126950 6178 heartbeater.cc:511] Master 127.0.98.62:43851 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 01:02:33.128468 5987 ts_manager.cc:194] Registered new tserver with Master: 53407a78f69746f5bf30346ba9143754 (127.0.98.1:38897)
I20260929 01:02:33.129590 5987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 01:02:33.144074 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43851
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:33.250705 6183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:33.251027 6183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:33.251070 6183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:33.255053 6183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:33.255102 6183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:33.255155 6183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:02:33.260237 6183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:33.261207 6183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:33.262239 6183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:33.271452 6191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:33.271463 6188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:33.271466 6189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:33.271524 6183 server_base.cc:1034] running on GCE node
I20260929 01:02:33.272173 6183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:33.272750 6183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:33.273949 6183 hybrid_clock.cc:648] HybridClock initialized: now 1790643753273932 us; error 52 us; skew 500 ppm
Sep 29 01:02:33 dist-test-slave-7zdl krb5kdc[5942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:33.276648 6183 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:02:33.277401 6183 webserver.cc:460] Webserver started at http://127.0.98.2:40387/ using document root <none> and password file <none>
I20260929 01:02:33.277920 6183 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:33.277980 6183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:33.278136 6183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:33.280055 6183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "7e78565a943b4a66921e56701644c630"
format_stamp: "Formatted at 2026-09-29 01:02:33 on dist-test-slave-7zdl"
server_key: "22445a1f217afb10771297270e3c267f"
server_key_iv: "56ac1787e0d8dede6e8bd95385695a2e"
server_key_version: "encryptionkey@0"
I20260929 01:02:33.280571 6183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7e78565a943b4a66921e56701644c630"
format_stamp: "Formatted at 2026-09-29 01:02:33 on dist-test-slave-7zdl"
server_key: "22445a1f217afb10771297270e3c267f"
server_key_iv: "56ac1787e0d8dede6e8bd95385695a2e"
server_key_version: "encryptionkey@0"
I20260929 01:02:33.283998 6183 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:02:33.286254 6198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:33.287452 6183 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:33.287578 6183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "7e78565a943b4a66921e56701644c630"
format_stamp: "Formatted at 2026-09-29 01:02:33 on dist-test-slave-7zdl"
server_key: "22445a1f217afb10771297270e3c267f"
server_key_iv: "56ac1787e0d8dede6e8bd95385695a2e"
server_key_version: "encryptionkey@0"
I20260929 01:02:33.287680 6183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:33.305452 6183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:33.308552 6183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:33.308740 6183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:33.309278 6183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:33.310050 6183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:33.310115 6183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:33.310181 6183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:33.310230 6183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:33.318557 6183 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40469
I20260929 01:02:33.318627 6311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40469 every 8 connection(s)
I20260929 01:02:33.319427 6183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:02:33.319702 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 6183
I20260929 01:02:33.319833 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:02:33.320165 392 external_mini_cluster.cc:1428] Setting key 086e70350b50d13a5d38bd0d24160c55
Sep 29 01:02:33 dist-test-slave-7zdl krb5kdc[5942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 01:02:33.332980 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:33.321314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43697 (local address 127.0.98.62:43851)
0929 01:02:33.321445 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:33.321449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:33.322431 (+ 982us) server_negotiation.cc:407] Connection header received
0929 01:02:33.323820 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:33.323823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:33.323872 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:33.323951 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:33.325023 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:33.325545 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:33.326636 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:33.326925 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:33.327033 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:33.329368 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:33.329371 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:33.329374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:33.329410 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:33.330879 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:33.331378 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:33.331382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:33.331396 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:33.331469 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:33.331816 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:33.331819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:33.331821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:33.331992 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:33.332092 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:33.332558 (+ 466us) server_negotiation.cc:299] Negotiation successful
0929 01:02:33.332674 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:02:33.333674 6314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:33.321824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43851 (local address 127.0.98.2:43697)
0929 01:02:33.322302 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:33.322340 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:33.323630 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:33.324122 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:33.324129 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:33.324450 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:33.324879 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:33.324900 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:33.325664 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:33.325668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:33.326516 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:33.326525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:33.327063 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:33.327067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:33.327234 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:33.327811 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:33.327826 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:33.329209 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:33.331004 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:33.331010 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:33.331023 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:33.331267 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:33.331558 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:33.331563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:33.331565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:33.331697 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:33.332100 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:33.332109 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:33.332334 (+ 225us) client_negotiation.cc:769] Sending connection context
0929 01:02:33.332506 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 01:02:33.332807 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":146,"threads_started":1}
I20260929 01:02:33.334832 6312 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43851
I20260929 01:02:33.335064 6312 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:33.335533 6312 heartbeater.cc:511] Master 127.0.98.62:43851 requested a full tablet report, sending...
I20260929 01:02:33.336573 5987 ts_manager.cc:194] Registered new tserver with Master: 7e78565a943b4a66921e56701644c630 (127.0.98.2:40469)
I20260929 01:02:33.337386 5987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 01:02:33.346159 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43851
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:33.453594 6317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:33.453830 6317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:33.453871 6317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:33.457927 6317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:33.457980 6317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:33.458086 6317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:02:33.463245 6317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:33.464221 6317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:33.465211 6317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:33.474424 6323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:33.474391 6325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:33.474380 6322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:33.474745 6317 server_base.cc:1034] running on GCE node
I20260929 01:02:33.475145 6317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:33.475767 6317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:33.476918 6317 hybrid_clock.cc:648] HybridClock initialized: now 1790643753476889 us; error 42 us; skew 500 ppm
Sep 29 01:02:33 dist-test-slave-7zdl krb5kdc[5942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:33.479653 6317 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:02:33.480412 6317 webserver.cc:460] Webserver started at http://127.0.98.3:36743/ using document root <none> and password file <none>
I20260929 01:02:33.480902 6317 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:33.480957 6317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:33.481106 6317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:33.483045 6317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "3251094639a04b8facd2feaf383dce4e"
format_stamp: "Formatted at 2026-09-29 01:02:33 on dist-test-slave-7zdl"
server_key: "7815c7dda94df01394e40916847d0290"
server_key_iv: "94782b59eca702476b9892d1f359c58c"
server_key_version: "encryptionkey@0"
I20260929 01:02:33.483533 6317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3251094639a04b8facd2feaf383dce4e"
format_stamp: "Formatted at 2026-09-29 01:02:33 on dist-test-slave-7zdl"
server_key: "7815c7dda94df01394e40916847d0290"
server_key_iv: "94782b59eca702476b9892d1f359c58c"
server_key_version: "encryptionkey@0"
I20260929 01:02:33.486930 6317 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:02:33.489115 6332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:33.490227 6317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:33.490316 6317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "3251094639a04b8facd2feaf383dce4e"
format_stamp: "Formatted at 2026-09-29 01:02:33 on dist-test-slave-7zdl"
server_key: "7815c7dda94df01394e40916847d0290"
server_key_iv: "94782b59eca702476b9892d1f359c58c"
server_key_version: "encryptionkey@0"
I20260929 01:02:33.490397 6317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:33.527529 6317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:33.530736 6317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:33.530941 6317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:33.531488 6317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:33.532239 6317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:33.532284 6317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:33.532359 6317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:33.532404 6317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:33.540963 6317 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34801
I20260929 01:02:33.540987 6445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34801 every 8 connection(s)
I20260929 01:02:33.541790 6317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:02:33.542217 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 6317
I20260929 01:02:33.542316 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:02:33.542605 392 external_mini_cluster.cc:1428] Setting key 523fedf78367da39bece233cae5728ba
Sep 29 01:02:33 dist-test-slave-7zdl krb5kdc[5942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:33.554330 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:33.543639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48859 (local address 127.0.98.62:43851)
0929 01:02:33.543754 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:33.543758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:33.544703 (+ 945us) server_negotiation.cc:407] Connection header received
0929 01:02:33.546111 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:33.546114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:33.546156 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:33.546227 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:33.547366 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:33.547885 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:33.548908 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:33.549119 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:33.549191 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:33.551514 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:33.551517 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:33.551519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:33.551541 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:33.552711 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:33.553133 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:33.553135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:33.553137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:33.553207 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:33.553484 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:33.553486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:33.553487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:33.553640 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:33.553711 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:33.554135 (+ 424us) server_negotiation.cc:299] Negotiation successful
0929 01:02:33.554218 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 01:02:33.555194 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:33.544081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43851 (local address 127.0.98.3:48859)
0929 01:02:33.544552 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:33.544624 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:33.545926 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:33.546380 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:33.546388 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:33.546714 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:33.547250 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:33.547263 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:33.547983 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:33.547985 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:33.548809 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:33.548817 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:33.549234 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:33.549236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:33.549366 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:33.549951 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:33.549968 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:33.551365 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:33.552804 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:33.552808 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:33.552822 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:33.553051 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:33.553280 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:33.553282 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:33.553284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:33.553411 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:33.553723 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:33.553727 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:33.553924 (+ 197us) client_negotiation.cc:769] Sending connection context
0929 01:02:33.554089 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 01:02:33.554310 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":137,"threads_started":1}
I20260929 01:02:33.556288 6446 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43851
I20260929 01:02:33.556559 6446 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:33.557015 6446 heartbeater.cc:511] Master 127.0.98.62:43851 requested a full tablet report, sending...
I20260929 01:02:33.557911 5987 ts_manager.cc:194] Registered new tserver with Master: 3251094639a04b8facd2feaf383dce4e (127.0.98.3:34801)
I20260929 01:02:33.558400 5987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48859
I20260929 01:02:33.567965 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/test-admin.keytab.
Sep 29 01:02:33 dist-test-slave-7zdl krb5kdc[5942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:33.583334 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 01:02:33.585924 6453 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:33.584575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39648 (local address 127.0.98.1:38897)
0929 01:02:33.584888 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:33.584902 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:33.584924 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:02:33.584971 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:33.584976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:33.585158 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:33.585282 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:33.585723 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39648: BlockingRecv error: recv got EOF from 127.0.0.1:39648 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":133,"threads_started":1}
W20260929 01:02:33.586011 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 01:02:33 dist-test-slave-7zdl krb5kdc[5942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:33.587047 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 01:02:33 dist-test-slave-7zdl krb5kdc[5942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:33.587955 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 01:02:33 dist-test-slave-7zdl krb5kdc[5942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 01:02:33.597188 6453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:33.588714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39654 (local address 127.0.98.1:38897)
0929 01:02:33.588823 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:33.588826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:33.588925 (+ 99us) server_negotiation.cc:407] Connection header received
0929 01:02:33.589014 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:33.589015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:33.589049 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:33.589106 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:33.589524 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:33.590279 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:33.591284 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:33.591791 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:33.591877 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:33.594130 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:33.594139 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:33.594150 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:33.594171 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:33.595657 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:33.596052 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:33.596056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:33.596058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:33.596127 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:33.596346 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:33.596348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:33.596349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:33.596564 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:33.596711 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:33.596849 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 01:02:33.597003 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 01:02:33.600638 6453 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:33.599786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39656 (local address 127.0.98.1:38897)
0929 01:02:33.599896 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:33.599902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:33.600008 (+ 106us) server_negotiation.cc:407] Connection header received
0929 01:02:33.600105 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:33.600107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:33.600140 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:33.600194 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:33.600555 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39656: BlockingRecv error: recv got EOF from 127.0.0.1:39656 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 01:02:33.600910 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 01:02:33 dist-test-slave-7zdl krb5kdc[5942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:33.601846 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 01:02:33 dist-test-slave-7zdl krb5kdc[5942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:33.602667 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 01:02:33 dist-test-slave-7zdl krb5kdc[5942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 01:02:33.610445 6453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:33.603425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39662 (local address 127.0.98.1:38897)
0929 01:02:33.603562 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:33.603567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:33.603632 (+ 65us) server_negotiation.cc:407] Connection header received
0929 01:02:33.603709 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:33.603711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:33.603741 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:33.603800 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:33.604234 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:33.604735 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:33.605484 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:33.605696 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:33.605764 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:33.607861 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:33.607864 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:33.607866 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:33.607887 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:33.609092 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:33.609485 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:33.609487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:33.609489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:33.609554 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:33.609764 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:33.609766 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:33.609767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:33.609900 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:33.609998 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:33.610174 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 01:02:33.610288 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 01:02:33.612978 6453 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:33.612023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.0.98.1:38897)
0929 01:02:33.612142 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:33.612146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:33.612257 (+ 111us) server_negotiation.cc:407] Connection header received
0929 01:02:33.612403 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:33.612406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:33.612445 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:33.612524 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:33.612887 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39668: BlockingRecv error: recv got EOF from 127.0.0.1:39668 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 01:02:33.613118 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790643720886468-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 01:02:33 dist-test-slave-7zdl krb5kdc[5942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:33.614041 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 01:02:33 dist-test-slave-7zdl krb5kdc[5942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:33.614981 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 01:02:33 dist-test-slave-7zdl krb5kdc[5942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 01:02:33.622406 6453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:33.615705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39674 (local address 127.0.98.1:38897)
0929 01:02:33.615842 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:33.615846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:33.615859 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:02:33.615981 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:33.615984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:33.616020 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:33.616091 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:33.616500 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:33.616943 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:33.617660 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:33.617850 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:33.617919 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:33.620003 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:33.620006 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:33.620008 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:33.620027 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:33.621133 (+ 1106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:33.621482 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:33.621485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:33.621486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:33.621548 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:33.621752 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:33.621754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:33.621755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:33.621880 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:33.621978 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:33.622139 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 01:02:33.622246 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 01:02:33.623333 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 6049
I20260929 01:02:33.628803 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 6183
I20260929 01:02:33.634261 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 6317
I20260929 01:02:33.639595 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 5955
2026-09-29T01:02:33Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2128 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:02:33 dist-test-slave-7zdl krb5kdc[6466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:02:33 dist-test-slave-7zdl krb5kdc[6466](info): set up 2 sockets
Sep 29 01:02:33 dist-test-slave-7zdl krb5kdc[6466](info): commencing operation
krb5kdc: starting...
W20260929 01:02:34.675274 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:02:34 dist-test-slave-7zdl krb5kdc[6466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:02:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:02:34Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260929 01:02:34.791800 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44771
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:34.898933 6479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:34.899222 6479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:34.899294 6479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:34.903308 6479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:02:34.903388 6479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:34.903427 6479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:34.903484 6479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:02:34.903520 6479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:02:34.908753 6479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:34.910000 6479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:34.911075 6479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:34.919130 6485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:34.919140 6484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:34.919137 6487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:34.919190 6479 server_base.cc:1034] running on GCE node
I20260929 01:02:34.919837 6479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:34.920863 6479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:34.922019 6479 hybrid_clock.cc:648] HybridClock initialized: now 1790643754921992 us; error 40 us; skew 500 ppm
Sep 29 01:02:34 dist-test-slave-7zdl krb5kdc[6466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643754, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:34.924713 6479 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260929 01:02:34.925565 6479 webserver.cc:460] Webserver started at http://127.0.98.62:45857/ using document root <none> and password file <none>
I20260929 01:02:34.926086 6479 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:34.926142 6479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:34.926384 6479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:34.928342 6479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "1d621ac6ce2e468f8451220161009672"
format_stamp: "Formatted at 2026-09-29 01:02:34 on dist-test-slave-7zdl"
server_key: "fd81449a06b03d8605634f5f671eb33a"
server_key_iv: "be05c898ae4d0c79f78391f9cbd214cb"
server_key_version: "encryptionkey@0"
I20260929 01:02:34.928884 6479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "1d621ac6ce2e468f8451220161009672"
format_stamp: "Formatted at 2026-09-29 01:02:34 on dist-test-slave-7zdl"
server_key: "fd81449a06b03d8605634f5f671eb33a"
server_key_iv: "be05c898ae4d0c79f78391f9cbd214cb"
server_key_version: "encryptionkey@0"
I20260929 01:02:34.932394 6479 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:02:34.934603 6494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:34.935773 6479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:34.935894 6479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "1d621ac6ce2e468f8451220161009672"
format_stamp: "Formatted at 2026-09-29 01:02:34 on dist-test-slave-7zdl"
server_key: "fd81449a06b03d8605634f5f671eb33a"
server_key_iv: "be05c898ae4d0c79f78391f9cbd214cb"
server_key_version: "encryptionkey@0"
I20260929 01:02:34.935993 6479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:34.946511 6479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:34.949602 6479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:34.949793 6479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:34.956629 6479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44771
I20260929 01:02:34.956887 6546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44771 every 8 connection(s)
I20260929 01:02:34.957551 6479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:02:34.960234 6547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:34.965369 6547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672: Bootstrap starting.
I20260929 01:02:34.967299 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 6479
I20260929 01:02:34.967413 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:02:34.967684 392 external_mini_cluster.cc:1428] Setting key d7ab6eb02c9a17ac2f4965754d349910
I20260929 01:02:34.967898 6547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:34.968739 6547 log.cc:824] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:34.970700 6547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672: No bootstrap required, opened a new log
I20260929 01:02:34.974723 6547 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } }
I20260929 01:02:34.975023 6547 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:34.975095 6547 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d621ac6ce2e468f8451220161009672, State: Initialized, Role: FOLLOWER
I20260929 01:02:34.975606 6547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } }
I20260929 01:02:34.975771 6547 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:02:34.975858 6547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:02:34.975950 6547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 01:02:34 dist-test-slave-7zdl krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 01:02:34.976830 6547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } }
I20260929 01:02:34.977156 6547 leader_election.cc:302] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d621ac6ce2e468f8451220161009672; no voters:
I20260929 01:02:34.977478 6547 leader_election.cc:288] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:02:34.977553 6552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:34.977777 6552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [term 1 LEADER]: Becoming Leader. State: Replica: 1d621ac6ce2e468f8451220161009672, State: Running, Role: LEADER
I20260929 01:02:34.978061 6552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } }
I20260929 01:02:34.978706 6547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:02:34.979254 6552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d621ac6ce2e468f8451220161009672. Latest consensus state: current_term: 1 leader_uuid: "1d621ac6ce2e468f8451220161009672" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } }
I20260929 01:02:34.979454 6552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:34.979662 6553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d621ac6ce2e468f8451220161009672" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } }
I20260929 01:02:34.979836 6553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:34.979876 6560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:02:34.982525 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:34.968750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33492 (local address 127.0.98.62:44771)
0929 01:02:34.969190 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:34.969204 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:34.969237 (+ 33us) server_negotiation.cc:407] Connection header received
0929 01:02:34.969838 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:34.969853 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:34.970150 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:34.970419 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:34.971314 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:34.972252 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:34.973436 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:34.973902 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:34.974015 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:34.976806 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:34.976811 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:34.976829 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:34.976860 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:34.978949 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:34.979501 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:34.979509 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:34.979517 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:34.979669 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:34.980003 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:34.980006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:34.980009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:34.980481 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:34.981200 (+ 719us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:34.981338 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 01:02:34.981623 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":152,"threads_started":1}
I20260929 01:02:34.983893 6560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:02:34.989598 6560 catalog_manager.cc:1347] Generated new cluster ID: 3792c0e2458b4d2088777c90f9a31006
I20260929 01:02:34.989658 6560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:02:35.006109 6560 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:02:35.007416 6560 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:02:35.022475 6560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672: Generated new TSK 0
I20260929 01:02:35.023306 6560 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260929 01:02:35.051578 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44771
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:35.163833 6573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:35.164114 6573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:35.164199 6573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:35.168174 6573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:35.168236 6573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:35.168370 6573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:02:35.173511 6573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:35.174566 6573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:35.175643 6573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:35.185654 6581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:35.185655 6578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:35.185683 6579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:35.185822 6573 server_base.cc:1034] running on GCE node
I20260929 01:02:35.186301 6573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:35.186998 6573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:35.188182 6573 hybrid_clock.cc:648] HybridClock initialized: now 1790643755188165 us; error 29 us; skew 500 ppm
Sep 29 01:02:35 dist-test-slave-7zdl krb5kdc[6466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643755, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:35.190752 6573 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 01:02:35.191613 6573 webserver.cc:460] Webserver started at http://127.0.98.1:36787/ using document root <none> and password file <none>
I20260929 01:02:35.192158 6573 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:35.192246 6573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:35.192478 6573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:35.194445 6573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23"
format_stamp: "Formatted at 2026-09-29 01:02:35 on dist-test-slave-7zdl"
server_key: "15433554939ea8ee96b3e8d8229076ec"
server_key_iv: "eeec67d802311488abbfd6308d58226d"
server_key_version: "encryptionkey@0"
I20260929 01:02:35.195047 6573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23"
format_stamp: "Formatted at 2026-09-29 01:02:35 on dist-test-slave-7zdl"
server_key: "15433554939ea8ee96b3e8d8229076ec"
server_key_iv: "eeec67d802311488abbfd6308d58226d"
server_key_version: "encryptionkey@0"
I20260929 01:02:35.198619 6573 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:02:35.200923 6588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:35.202248 6573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:35.202375 6573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23"
format_stamp: "Formatted at 2026-09-29 01:02:35 on dist-test-slave-7zdl"
server_key: "15433554939ea8ee96b3e8d8229076ec"
server_key_iv: "eeec67d802311488abbfd6308d58226d"
server_key_version: "encryptionkey@0"
I20260929 01:02:35.202479 6573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:35.233603 6573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:35.236495 6573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:35.236696 6573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:35.237284 6573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:35.238093 6573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:35.238166 6573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:35.238236 6573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:35.238291 6573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:35.247112 6573 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36813
I20260929 01:02:35.247138 6701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36813 every 8 connection(s)
I20260929 01:02:35.247970 6573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
Sep 29 01:02:35 dist-test-slave-7zdl krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643755, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 01:02:35.257802 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 6573
I20260929 01:02:35.257933 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:02:35.258293 392 external_mini_cluster.cc:1428] Setting key 3f691f7eb9b482c4bc99c2f208ba5cc6
I20260929 01:02:35.261643 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:35.249767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45431 (local address 127.0.98.62:44771)
0929 01:02:35.249906 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:35.249910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:35.250703 (+ 793us) server_negotiation.cc:407] Connection header received
0929 01:02:35.252010 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:35.252012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:35.252057 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:35.252152 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:35.253252 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.253824 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.254949 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.255174 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.255260 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:35.257611 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:35.257616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:35.257620 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:35.257651 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:35.259430 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:35.260002 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:35.260006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:35.260008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:35.260095 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:35.260437 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:35.260443 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:35.260448 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:35.260708 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:35.260809 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:35.261411 (+ 602us) server_negotiation.cc:299] Negotiation successful
0929 01:02:35.261525 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 01:02:35.263103 6704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:35.250136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.98.1:45431)
0929 01:02:35.250557 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:35.250599 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:35.251823 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:35.252281 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:35.252288 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:35.252633 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:35.253097 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:35.253109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.253947 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.253951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:35.254846 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:35.254855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.255301 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.255304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:35.255438 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:35.255962 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:35.255978 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:35.257422 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:35.259545 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:35.259550 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:35.259570 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:35.259907 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:35.260186 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:35.260189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:35.260190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:35.260338 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:35.260804 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:35.260808 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:35.261126 (+ 318us) client_negotiation.cc:769] Sending connection context
0929 01:02:35.261824 (+ 698us) client_negotiation.cc:240] Negotiation successful
0929 01:02:35.262121 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":98,"threads_started":1}
I20260929 01:02:35.264492 6702 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44771
I20260929 01:02:35.264725 6702 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:35.265198 6702 heartbeater.cc:511] Master 127.0.98.62:44771 requested a full tablet report, sending...
I20260929 01:02:35.266656 6511 ts_manager.cc:194] Registered new tserver with Master: fb1e8fd0fcac4b96bf8d5c75404d2e23 (127.0.98.1:36813)
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 01:02:35.267779 6511 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260929 01:02:35.285378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44771
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:35.393174 6707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:35.393416 6707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:35.393458 6707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:35.397574 6707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:35.397635 6707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:35.397694 6707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:02:35.402874 6707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:35.403820 6707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:35.404775 6707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:35.414145 6715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:35.414196 6712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:35.414422 6713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:35.414412 6707 server_base.cc:1034] running on GCE node
I20260929 01:02:35.414976 6707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:35.415562 6707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:35.416718 6707 hybrid_clock.cc:648] HybridClock initialized: now 1790643755416704 us; error 26 us; skew 500 ppm
Sep 29 01:02:35 dist-test-slave-7zdl krb5kdc[6466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643755, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:35.419337 6707 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 01:02:35.420094 6707 webserver.cc:460] Webserver started at http://127.0.98.2:40497/ using document root <none> and password file <none>
I20260929 01:02:35.420600 6707 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:35.420655 6707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:35.420809 6707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:35.422693 6707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "d158538b2a3d427182b72f8694c1e589"
format_stamp: "Formatted at 2026-09-29 01:02:35 on dist-test-slave-7zdl"
server_key: "12cb589c1a78882abdbf8c9a1eb97dd8"
server_key_iv: "4c23d6aba532bad0f33c89aeec21902a"
server_key_version: "encryptionkey@0"
I20260929 01:02:35.423249 6707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d158538b2a3d427182b72f8694c1e589"
format_stamp: "Formatted at 2026-09-29 01:02:35 on dist-test-slave-7zdl"
server_key: "12cb589c1a78882abdbf8c9a1eb97dd8"
server_key_iv: "4c23d6aba532bad0f33c89aeec21902a"
server_key_version: "encryptionkey@0"
I20260929 01:02:35.426607 6707 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:02:35.428853 6722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:35.430027 6707 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 01:02:35.430150 6707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "d158538b2a3d427182b72f8694c1e589"
format_stamp: "Formatted at 2026-09-29 01:02:35 on dist-test-slave-7zdl"
server_key: "12cb589c1a78882abdbf8c9a1eb97dd8"
server_key_iv: "4c23d6aba532bad0f33c89aeec21902a"
server_key_version: "encryptionkey@0"
I20260929 01:02:35.430248 6707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:35.445485 6707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:35.448137 6707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:35.448329 6707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:35.448877 6707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:35.449672 6707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:35.449745 6707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:35.449811 6707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:35.449844 6707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:35.458623 6707 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42313
I20260929 01:02:35.458653 6835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42313 every 8 connection(s)
I20260929 01:02:35.459512 6707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:02:35.461063 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 6707
I20260929 01:02:35.461179 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:02:35.461499 392 external_mini_cluster.cc:1428] Setting key 38e172b63052a2009795a6b0349357f2
Sep 29 01:02:35 dist-test-slave-7zdl krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643755, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 01:02:35.473122 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:35.461224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32871 (local address 127.0.98.62:44771)
0929 01:02:35.461389 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:35.461394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:35.462224 (+ 830us) server_negotiation.cc:407] Connection header received
0929 01:02:35.463672 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:35.463679 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:35.463740 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:35.463830 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:35.464923 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.465669 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.466752 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.467038 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.467149 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:35.469542 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:35.469549 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:35.469554 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:35.469587 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:35.471242 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:35.471702 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:35.471704 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:35.471706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:35.471777 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:35.472055 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:35.472057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:35.472058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:35.472272 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:35.472348 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:35.472797 (+ 449us) server_negotiation.cc:299] Negotiation successful
0929 01:02:35.472889 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260929 01:02:35.474033 6838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:35.461636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.98.2:32871)
0929 01:02:35.462083 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:35.462134 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:35.463466 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:35.464006 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:35.464012 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:35.464337 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:35.464783 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:35.464795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.465776 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.465779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:35.466628 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:35.466637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.467180 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.467184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:35.467356 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:35.467976 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:35.467991 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:35.469367 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:35.471351 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:35.471355 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:35.471367 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:35.471615 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:35.471854 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:35.471856 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:35.471857 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:35.471978 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:35.472356 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:35.472361 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:35.472590 (+ 229us) client_negotiation.cc:769] Sending connection context
0929 01:02:35.472753 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 01:02:35.473040 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":124,"threads_started":1}
I20260929 01:02:35.475250 6836 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44771
I20260929 01:02:35.475507 6836 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:35.475991 6836 heartbeater.cc:511] Master 127.0.98.62:44771 requested a full tablet report, sending...
I20260929 01:02:35.477022 6511 ts_manager.cc:194] Registered new tserver with Master: d158538b2a3d427182b72f8694c1e589 (127.0.98.2:42313)
I20260929 01:02:35.477656 6511 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:32871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 01:02:35.489714 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44771
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:35.596684 6841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:35.596940 6841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:35.597023 6841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:35.601028 6841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:35.601084 6841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:35.601140 6841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:02:35.606240 6841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:35.607275 6841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:35.608209 6841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:35.618021 6846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:35.618021 6847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:35.618117 6849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:35.618258 6841 server_base.cc:1034] running on GCE node
I20260929 01:02:35.618763 6841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:35.619503 6841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:35.620679 6841 hybrid_clock.cc:648] HybridClock initialized: now 1790643755620661 us; error 32 us; skew 500 ppm
Sep 29 01:02:35 dist-test-slave-7zdl krb5kdc[6466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643755, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:35.623489 6841 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 01:02:35.624424 6841 webserver.cc:460] Webserver started at http://127.0.98.3:42459/ using document root <none> and password file <none>
I20260929 01:02:35.624998 6841 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:35.625085 6841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:35.625314 6841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:35.627727 6841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "1555431512ad4290b1cd00b2a315816e"
format_stamp: "Formatted at 2026-09-29 01:02:35 on dist-test-slave-7zdl"
server_key: "fece09d6e1a850524f7b3a279c3308f4"
server_key_iv: "fa1fef084ab3490021a40a6e868693db"
server_key_version: "encryptionkey@0"
I20260929 01:02:35.628304 6841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1555431512ad4290b1cd00b2a315816e"
format_stamp: "Formatted at 2026-09-29 01:02:35 on dist-test-slave-7zdl"
server_key: "fece09d6e1a850524f7b3a279c3308f4"
server_key_iv: "fa1fef084ab3490021a40a6e868693db"
server_key_version: "encryptionkey@0"
I20260929 01:02:35.632305 6841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:02:35.635165 6856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:35.636459 6841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:02:35.636602 6841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "1555431512ad4290b1cd00b2a315816e"
format_stamp: "Formatted at 2026-09-29 01:02:35 on dist-test-slave-7zdl"
server_key: "fece09d6e1a850524f7b3a279c3308f4"
server_key_iv: "fa1fef084ab3490021a40a6e868693db"
server_key_version: "encryptionkey@0"
I20260929 01:02:35.636718 6841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:35.662029 6841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:35.665679 6841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:35.665902 6841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:35.666515 6841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:35.667425 6841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:35.667502 6841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:35.667577 6841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:35.667632 6841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:35.677939 6841 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34461
I20260929 01:02:35.677969 6969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34461 every 8 connection(s)
I20260929 01:02:35.678951 6841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:02:35.685491 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 6841
I20260929 01:02:35.685631 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:02:35.685912 392 external_mini_cluster.cc:1428] Setting key d4e423fccb827a786551100db61922de
Sep 29 01:02:35 dist-test-slave-7zdl krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643755, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 01:02:35.692814 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:35.681015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44777 (local address 127.0.98.62:44771)
0929 01:02:35.681150 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:35.681154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:35.681966 (+ 812us) server_negotiation.cc:407] Connection header received
0929 01:02:35.683336 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:35.683339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:35.683398 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:35.683497 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:35.684647 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.685187 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.686522 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.686870 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.686986 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:35.689547 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:35.689550 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:35.689553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:35.689575 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:35.691108 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:35.691540 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:35.691542 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:35.691543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:35.691616 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:35.691891 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:35.691893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:35.691895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:35.692054 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:35.692127 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:35.692582 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 01:02:35.692694 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 01:02:35.693606 6972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:35.681375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.98.3:44777)
0929 01:02:35.681811 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:35.681852 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:35.683124 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:35.683665 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:35.683672 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:35.684041 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:35.684501 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:35.684513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.685311 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.685315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:35.686375 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:35.686386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.687040 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.687045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:35.687203 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:35.687823 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:35.687843 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:35.689398 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:35.691200 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:35.691204 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:35.691215 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:35.691451 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:35.691694 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:35.691696 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:35.691697 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:35.691818 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:35.692135 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:35.692139 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:35.692354 (+ 215us) client_negotiation.cc:769] Sending connection context
0929 01:02:35.692502 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 01:02:35.692741 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":114,"threads_started":1}
I20260929 01:02:35.694708 6970 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44771
I20260929 01:02:35.694978 6970 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:35.695479 6970 heartbeater.cc:511] Master 127.0.98.62:44771 requested a full tablet report, sending...
I20260929 01:02:35.696508 6511 ts_manager.cc:194] Registered new tserver with Master: 1555431512ad4290b1cd00b2a315816e (127.0.98.3:34461)
I20260929 01:02:35.697043 6511 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44777
I20260929 01:02:35.699944 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260929 01:02:35.729549 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40081
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44771,127.0.98.61:40081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:35.838145 6975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:35.838441 6975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:35.838529 6975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:35.842649 6975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:02:35.842728 6975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:35.842765 6975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:35.842839 6975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:02:35.842898 6975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:02:35.847981 6975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44771,127.0.98.61:40081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
W20260929 01:02:35.848210 6975 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:44771,127.0.98.61:40081'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 01:02:35.849044 6975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:35.850030 6975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:35.858273 6980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:35.858258 6983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:35.858347 6975 server_base.cc:1034] running on GCE node
W20260929 01:02:35.858244 6981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:35.859115 6975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:35.860144 6975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:35.861323 6975 hybrid_clock.cc:648] HybridClock initialized: now 1790643755861309 us; error 24 us; skew 500 ppm
Sep 29 01:02:35 dist-test-slave-7zdl krb5kdc[6466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643755, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:35.863934 6975 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260929 01:02:35.864795 6975 webserver.cc:460] Webserver started at http://127.0.98.61:41827/ using document root <none> and password file <none>
I20260929 01:02:35.865319 6975 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:35.865420 6975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:35.865646 6975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:35.867918 6975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/data/instance:
uuid: "7bcff1ee79b34e0cb34c599ba53620e9"
format_stamp: "Formatted at 2026-09-29 01:02:35 on dist-test-slave-7zdl"
server_key: "503835515277a488a4f3c20484e52f3e"
server_key_iv: "ace3fb1441573c8384eb2652f005479f"
server_key_version: "encryptionkey@0"
I20260929 01:02:35.868515 6975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/wal/instance:
uuid: "7bcff1ee79b34e0cb34c599ba53620e9"
format_stamp: "Formatted at 2026-09-29 01:02:35 on dist-test-slave-7zdl"
server_key: "503835515277a488a4f3c20484e52f3e"
server_key_iv: "ace3fb1441573c8384eb2652f005479f"
server_key_version: "encryptionkey@0"
I20260929 01:02:35.871973 6975 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:02:35.874164 6990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:35.875414 6975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:02:35.875550 6975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/wal
uuid: "7bcff1ee79b34e0cb34c599ba53620e9"
format_stamp: "Formatted at 2026-09-29 01:02:35 on dist-test-slave-7zdl"
server_key: "503835515277a488a4f3c20484e52f3e"
server_key_iv: "ace3fb1441573c8384eb2652f005479f"
server_key_version: "encryptionkey@0"
I20260929 01:02:35.875653 6975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:35.894250 6975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:35.897336 6975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:35.897553 6975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:35.904353 6975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40081
I20260929 01:02:35.904632 7042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40081 every 8 connection(s)
I20260929 01:02:35.905387 6975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/data/info.pb
I20260929 01:02:35.908201 7043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:35.911862 7043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:02:35.915407 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 6975
Sep 29 01:02:35 dist-test-slave-7zdl krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643755, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 01:02:35 dist-test-slave-7zdl krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 01:02:35.925633 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:35.913692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33494 (local address 127.0.98.62:44771)
0929 01:02:35.913831 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:35.913835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:35.914637 (+ 802us) server_negotiation.cc:407] Connection header received
0929 01:02:35.915654 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:35.915658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:35.915711 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:35.915817 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:35.917103 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.917640 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.918840 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.919070 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.919152 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:35.921773 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:35.921776 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:35.921779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:35.921801 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:35.923378 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:35.924004 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:35.924007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:35.924008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:35.924094 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:35.924463 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:35.924466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:35.924467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:35.924693 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:35.924823 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:35.925321 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 01:02:35.925446 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 01:02:35.926545 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:35.916206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58300 (local address 127.0.98.61:40081)
0929 01:02:35.916543 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:35.916555 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:35.916579 (+ 24us) server_negotiation.cc:407] Connection header received
0929 01:02:35.916652 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:35.916658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:35.916877 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:35.917043 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:35.917758 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.918654 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.919714 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.920183 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.920262 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:35.922665 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:35.922673 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:35.922688 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:35.922714 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:35.924551 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:35.924964 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:35.924971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:35.924973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:35.925065 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:35.925356 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:35.925359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:35.925361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:35.925582 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:35.925779 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:35.925962 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 01:02:35.926093 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":229,"spinlock_wait_cycles":896,"thread_start_us":137,"threads_started":1}
I20260929 01:02:35.926554 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:35.914042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.0.1:33494)
0929 01:02:35.914482 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:35.914524 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:35.915411 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:35.915983 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:35.915990 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:35.916483 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:35.916951 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:35.916964 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.917793 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.917796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:35.918658 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:35.918666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.919214 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.919218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:35.919412 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:35.920067 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:35.920082 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:35.921628 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:35.923493 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:35.923501 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:35.923510 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:35.923873 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:35.924193 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:35.924197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:35.924199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:35.924361 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:35.924819 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:35.924826 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:35.925065 (+ 239us) client_negotiation.cc:769] Sending connection context
0929 01:02:35.925272 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 01:02:35.925572 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":41,"thread_start_us":112,"threads_started":1}
I20260929 01:02:35.927977 7043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:02:35 dist-test-slave-7zdl krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643755, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 01:02:35.935650 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:35.928492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40081 (local address 127.0.0.1:58312)
0929 01:02:35.928626 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:35.928643 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:35.928732 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:35.928970 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:35.928973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:35.929118 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:35.929190 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:35.929195 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.929906 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.929912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:35.930473 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:35.930480 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.931051 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.931054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:35.931148 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:35.931508 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:35.931519 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:35.932739 (+ 1220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:35.934396 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:35.934398 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:35.934400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:35.934641 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:35.934929 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:35.934931 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:35.934932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:35.935013 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:35.935286 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:35.935288 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:35.935353 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 01:02:35.935427 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 01:02:35.935522 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 01:02:35.935681 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:35.928654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58312 (local address 127.0.98.61:40081)
0929 01:02:35.928797 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:35.928803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:35.928815 (+ 12us) server_negotiation.cc:407] Connection header received
0929 01:02:35.928841 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:35.928843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:35.928881 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:35.928961 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:35.929302 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.929796 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.930582 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.930900 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.931052 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:35.932854 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:35.932857 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:35.932859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:35.932889 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:35.934310 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:35.934706 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:35.934709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:35.934710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:35.934782 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:35.935072 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:35.935074 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:35.935075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:35.935215 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:35.935282 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:35.935459 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 01:02:35.935570 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":62}
I20260929 01:02:35.938591 7043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9: Bootstrap starting.
I20260929 01:02:35.941030 7043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:35.941721 7043 log.cc:824] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:35.943670 7043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9: No bootstrap required, opened a new log
I20260929 01:02:35.946484 7043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } }
I20260929 01:02:35.946702 7043 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:35.946802 7043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bcff1ee79b34e0cb34c599ba53620e9, State: Initialized, Role: FOLLOWER
I20260929 01:02:35.947278 7043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } }
I20260929 01:02:35.947687 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } } }
I20260929 01:02:35.947788 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:02:35.948357 7043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:02:35.952872 7060 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:02:35.952955 7060 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:02:35.954731 6975 master_runner.cc:428] Detected that this master 7bcff1ee79b34e0cb34c599ba53620e9 is joining an existing cluster
I20260929 01:02:35.954784 6975 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:40081
I20260929 01:02:35.966073 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:35.956835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58324 (local address 127.0.98.61:40081)
0929 01:02:35.956932 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:35.956937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:35.957659 (+ 722us) server_negotiation.cc:407] Connection header received
0929 01:02:35.957814 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:35.957818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:35.957854 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:35.957902 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:35.958561 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.959386 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.960594 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.960936 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.961005 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:35.962880 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:35.962884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:35.962887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:35.962908 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:35.964517 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:35.964941 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:35.964944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:35.964945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:35.965025 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:35.965275 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:35.965278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:35.965280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:35.965492 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:35.965591 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:35.965827 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 01:02:35.965926 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 01:02:35.966745 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:35.957143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40081 (local address 127.0.0.1:58324)
0929 01:02:35.957546 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:35.957560 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:35.957719 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:35.958018 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:35.958020 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:35.958261 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:35.958400 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:35.958406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.959598 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.959601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:35.960476 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:35.960486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.961146 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.961149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:35.961231 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:35.962279 (+ 1048us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:35.962299 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:35.962723 (+ 424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:35.964626 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:35.964628 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:35.964630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:35.964848 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:35.965121 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:35.965124 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:35.965126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:35.965195 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:35.965592 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:35.965595 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:35.965668 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 01:02:35.966468 (+ 800us) client_negotiation.cc:240] Negotiation successful
0929 01:02:35.966586 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":119,"threads_started":1}
I20260929 01:02:35.969086 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:35.960044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.0.1:33496)
0929 01:02:35.960336 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:35.960348 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:35.960483 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:35.960799 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:35.960801 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:35.961018 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:35.961176 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:35.961182 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.962489 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.962492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:35.963919 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:35.963929 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.964587 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.964590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:35.964689 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:35.965137 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:35.965150 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:35.965746 (+ 596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:35.967658 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:35.967661 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:35.967664 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:35.967876 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:35.968134 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:35.968137 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:35.968139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:35.968204 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:35.968629 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:35.968632 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:35.968716 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 01:02:35.968815 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 01:02:35.968929 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":87,"threads_started":1}
I20260929 01:02:35.969130 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:35.959825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33496 (local address 127.0.98.62:44771)
0929 01:02:35.959930 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:35.959934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:35.960423 (+ 489us) server_negotiation.cc:407] Connection header received
0929 01:02:35.960578 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:35.960581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:35.960619 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:35.960661 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:35.961298 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.962347 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.964056 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.964386 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.964450 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:35.965878 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:35.965882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:35.965885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:35.965908 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:35.967538 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:35.967971 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:35.967974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:35.967976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:35.968044 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:35.968302 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:35.968306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:35.968308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:35.968525 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:35.968622 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:35.968878 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 01:02:35.968989 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 01:02:35 dist-test-slave-7zdl krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643754, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 01:02:35.982671 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:35.973643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58338 (local address 127.0.98.61:40081)
0929 01:02:35.973737 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:35.973741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:35.974025 (+ 284us) server_negotiation.cc:407] Connection header received
0929 01:02:35.974191 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:35.974194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:35.974241 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:35.974320 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:35.974916 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.975652 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.976457 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.976780 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.976881 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:35.978960 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:35.978963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:35.978966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:35.979011 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:35.980746 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:35.981274 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:35.981278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:35.981280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:35.981353 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:35.981647 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:35.981650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:35.981652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:35.981838 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:35.981901 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:35.982341 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 01:02:35.982473 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 01:02:35.982887 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:35.973572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40081 (local address 127.0.0.1:58338)
0929 01:02:35.973899 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:35.973941 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:35.974067 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:35.974356 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:35.974360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:35.974671 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:35.974760 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:35.974765 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.975747 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.975749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:35.976344 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:35.976351 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:35.976896 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:35.976899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:35.976974 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:35.977442 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:35.977453 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:35.978778 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:35.980868 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:35.980877 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:35.980883 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:35.981163 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:35.981469 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:35.981472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:35.981474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:35.981553 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:35.981978 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:35.981985 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:35.982105 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 01:02:35.982256 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 01:02:35.982456 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":102,"threads_started":1}
I20260929 01:02:35.984474 6511 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:40081: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 40081
}
attrs {
promote: true
}
}
dest_uuid: "1d621ac6ce2e468f8451220161009672"
cas_config_opid_index: -1
I20260929 01:02:35.985149 6511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: true } }
I20260929 01:02:35.987852 7070 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7bcff1ee79b34e0cb34c599ba53620e9 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: true } } }
I20260929 01:02:35.988248 7017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [term 0 FOLLOWER]: Advancing to term 1
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260929 01:02:35.989606 7017 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [term 1 FOLLOWER]: Refusing update from remote peer 1d621ac6ce2e468f8451220161009672: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 01:02:35.989626 7070 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:40081
I20260929 01:02:35.989832 7070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1d621ac6ce2e468f8451220161009672" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: true } } }
I20260929 01:02:35.989918 7070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:35.990072 6975 master.cc:537] Master@127.0.98.61:40081 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260929 01:02:35.991696 7070 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:35.994618 6975 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:35.994978 6975 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:35.995157 6975 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9: stopping tablet replica
W20260929 01:02:35.995225 6496 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 01:02:35.997620 6496 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 -> Peer 7bcff1ee79b34e0cb34c599ba53620e9 (127.0.98.61:40081): Couldn't send request to peer 7bcff1ee79b34e0cb34c599ba53620e9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260929 01:02:36.010356 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36721
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44771,127.0.98.61:40081,127.0.98.60:36721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:02:36.011080 6975 master.cc:552] Master@127.0.98.61:40081 shutdown complete.
I20260929 01:02:36.011144 6975 master_runner.cc:305] Clearing existing system tablet
I20260929 01:02:36.011400 6975 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:36.011479 6975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:36.012904 6975 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 01:02:36.013904 7077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:36.014293 6975 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:36.014411 6975 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:36.014443 6975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/wal
uuid: "7bcff1ee79b34e0cb34c599ba53620e9"
format_stamp: "Formatted at 2026-09-29 01:02:35 on dist-test-slave-7zdl"
server_key: "503835515277a488a4f3c20484e52f3e"
server_key_iv: "ace3fb1441573c8384eb2652f005479f"
server_key_version: "encryptionkey@0"
I20260929 01:02:36.016733 6975 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:36.019129 6975 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 01:02:36.019233 6975 log.cc:1196] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 01:02:36.019493 6975 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9: Deleting consensus metadata
I20260929 01:02:36.019654 6975 master_runner.cc:315] Copying system tablet from 127.0.98.62:44771
I20260929 01:02:36.020850 6975 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:44771
I20260929 01:02:36.029917 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.021416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.0.1:33506)
0929 01:02:36.021686 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:36.021699 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:36.021811 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:36.022069 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:36.022072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:36.022270 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:36.022400 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.022406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.023491 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.023494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.024477 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.024487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.025086 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.025089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.025183 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.025766 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:36.025780 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:36.026191 (+ 411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:36.028310 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.028314 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.028316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.028568 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.028854 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.028856 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.028859 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.028943 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.029360 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:36.029363 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:36.029461 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 01:02:36.029567 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 01:02:36.029712 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":77,"threads_started":1}
I20260929 01:02:36.029947 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.021501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33506 (local address 127.0.98.62:44771)
0929 01:02:36.021623 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.021627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.021784 (+ 157us) server_negotiation.cc:407] Connection header received
0929 01:02:36.021893 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.021895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.021945 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.022042 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:36.022515 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.023383 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.024607 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.024983 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.025081 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.026339 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:36.026343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:36.026346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:36.026372 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:36.028189 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.028667 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.028671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.028673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.028759 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.029038 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.029041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.029043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.029252 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:36.029362 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.029634 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.029748 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 01:02:36.030867 6531 tablet_copy_service.cc:140] P 1d621ac6ce2e468f8451220161009672: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7bcff1ee79b34e0cb34c599ba53620e9 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33506)
I20260929 01:02:36.031056 6531 tablet_copy_service.cc:161] P 1d621ac6ce2e468f8451220161009672: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7bcff1ee79b34e0cb34c599ba53620e9 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33506: session id = 7bcff1ee79b34e0cb34c599ba53620e9-00000000000000000000000000000000
I20260929 01:02:36.032789 6531 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 01:02:36.034377 6975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 01:02:36.036734 6975 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9: tablet copy: Starting download of 0 data blocks...
I20260929 01:02:36.036913 6975 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9: tablet copy: Starting download of 1 WAL segments...
I20260929 01:02:36.038635 6975 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 01:02:36.040428 6531 tablet_copy_service.cc:342] P 1d621ac6ce2e468f8451220161009672: Request end of tablet copy session 7bcff1ee79b34e0cb34c599ba53620e9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33506
I20260929 01:02:36.040542 6531 tablet_copy_service.cc:433] P 1d621ac6ce2e468f8451220161009672: ending tablet copy session 7bcff1ee79b34e0cb34c599ba53620e9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7bcff1ee79b34e0cb34c599ba53620e9
I20260929 01:02:36.047453 6975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:36.049057 7084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:36.049058 7087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:36.049365 7085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:36.049633 6975 server_base.cc:1034] running on GCE node
I20260929 01:02:36.049773 6975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:36.050056 6975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:36.051190 6975 hybrid_clock.cc:648] HybridClock initialized: now 1790643756051169 us; error 36 us; skew 500 ppm
Sep 29 01:02:36 dist-test-slave-7zdl krb5kdc[6466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643756, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:36.052785 6975 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260929 01:02:36.053292 6975 webserver.cc:460] Webserver started at http://127.0.98.61:33459/ using document root <none> and password file <none>
I20260929 01:02:36.053479 6975 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:36.053520 6975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:36.059258 6975 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260929 01:02:36.060289 7094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:36.060524 6975 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:36.060596 6975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/wal
uuid: "7bcff1ee79b34e0cb34c599ba53620e9"
format_stamp: "Formatted at 2026-09-29 01:02:35 on dist-test-slave-7zdl"
server_key: "503835515277a488a4f3c20484e52f3e"
server_key_iv: "ace3fb1441573c8384eb2652f005479f"
server_key_version: "encryptionkey@0"
I20260929 01:02:36.060685 6975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:36.069682 6975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:36.071861 6975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:36.076649 6975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40081
I20260929 01:02:36.077310 6975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-1/data/info.pb
I20260929 01:02:36.077690 7146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40081 every 8 connection(s)
I20260929 01:02:36.079186 7147 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:02:36.080241 7147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9: Bootstrap starting.
I20260929 01:02:36.089937 7147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:02:36.090233 7147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9: Bootstrap complete.
I20260929 01:02:36.090838 7147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: true } }
I20260929 01:02:36.090982 7147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7bcff1ee79b34e0cb34c599ba53620e9, State: Initialized, Role: LEARNER
I20260929 01:02:36.091094 7147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: true } }
I20260929 01:02:36.091396 7150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: true } } }
I20260929 01:02:36.091571 7150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [sys.catalog]: This master's current role is: LEARNER
I20260929 01:02:36.091579 7147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:02:36.095455 7161 catalog_manager.cc:1259] Loaded cluster ID: 3792c0e2458b4d2088777c90f9a31006
I20260929 01:02:36.095525 7161 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9: loading cluster ID for follower catalog manager: success
I20260929 01:02:36.097568 7161 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9: acquiring CA information for follower catalog manager: success
I20260929 01:02:36.098948 7161 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:02:36.137799 7075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:36.138041 7075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:36.138110 7075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:36.142347 7075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:02:36.142405 7075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:36.142423 7075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:36.142463 7075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:02:36.142494 7075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:02:36.147673 7075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44771,127.0.98.61:40081,127.0.98.60:36721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:36.148645 7075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:36.149628 7075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:36.157869 7167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:36.157898 7169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:36.157931 7075 server_base.cc:1034] running on GCE node
W20260929 01:02:36.157898 7166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:36.158681 7075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:36.159818 7075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:36.160970 7075 hybrid_clock.cc:648] HybridClock initialized: now 1790643756160959 us; error 28 us; skew 500 ppm
Sep 29 01:02:36 dist-test-slave-7zdl krb5kdc[6466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643756, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:36.163668 7075 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260929 01:02:36.164546 7075 webserver.cc:460] Webserver started at http://127.0.98.60:44131/ using document root <none> and password file <none>
I20260929 01:02:36.165081 7075 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:36.165138 7075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:36.165367 7075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:36.167384 7075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/data/instance:
uuid: "1a6f4f441458470689115fea13835036"
format_stamp: "Formatted at 2026-09-29 01:02:36 on dist-test-slave-7zdl"
server_key: "65041022dffa16cce8166651bc14ec70"
server_key_iv: "657fd4d697bba53051aa8c685712880a"
server_key_version: "encryptionkey@0"
I20260929 01:02:36.167924 7075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/wal/instance:
uuid: "1a6f4f441458470689115fea13835036"
format_stamp: "Formatted at 2026-09-29 01:02:36 on dist-test-slave-7zdl"
server_key: "65041022dffa16cce8166651bc14ec70"
server_key_iv: "657fd4d697bba53051aa8c685712880a"
server_key_version: "encryptionkey@0"
I20260929 01:02:36.171398 7075 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:02:36.173575 7176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:36.174742 7075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:36.174914 7075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/wal
uuid: "1a6f4f441458470689115fea13835036"
format_stamp: "Formatted at 2026-09-29 01:02:36 on dist-test-slave-7zdl"
server_key: "65041022dffa16cce8166651bc14ec70"
server_key_iv: "657fd4d697bba53051aa8c685712880a"
server_key_version: "encryptionkey@0"
I20260929 01:02:36.175001 7075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:36.198006 7075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:36.201315 7075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:36.201462 7075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:36.208402 7075 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36721
I20260929 01:02:36.208663 7228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36721 every 8 connection(s)
I20260929 01:02:36.209372 7075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/data/info.pb
I20260929 01:02:36.212112 7229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:36.215794 7229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:02:36.217273 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 7075
Sep 29 01:02:36 dist-test-slave-7zdl krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643756, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 01:02:36 dist-test-slave-7zdl krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 01:02:36.228526 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.217626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33510 (local address 127.0.98.62:44771)
0929 01:02:36.217773 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.217777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.218626 (+ 849us) server_negotiation.cc:407] Connection header received
0929 01:02:36.219495 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.219499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.219553 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.219651 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:36.220415 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.220944 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.221952 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.222180 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.222290 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.224878 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:36.224882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:36.224887 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:36.224920 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:36.226441 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.226958 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.226962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.226964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.227050 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.227386 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.227391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.227393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.227582 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:36.227687 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.228244 (+ 557us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.228353 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 01:02:36.229322 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.218128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60042 (local address 127.0.98.60:36721)
0929 01:02:36.218574 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.218584 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.218621 (+ 37us) server_negotiation.cc:407] Connection header received
0929 01:02:36.219298 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.219315 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.219507 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.219686 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:36.220506 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.221430 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.222239 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.222614 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.222728 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.225428 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:36.225434 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:36.225448 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:36.225474 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:36.227298 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.227761 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.227766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.227768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.227838 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.228143 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.228146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.228148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.228413 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:36.228590 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.228799 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.228931 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":363,"spinlock_wait_cycles":51968,"thread_start_us":158,"threads_started":1}
I20260929 01:02:36.229322 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.217998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.0.1:33510)
0929 01:02:36.218495 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:36.218534 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:36.219312 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:36.219696 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:36.219702 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:36.220040 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:36.220271 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.220283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.221068 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.221073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.221825 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.221833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.222319 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.222322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.222505 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.223118 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:36.223143 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:36.224738 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:36.226542 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.226551 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.226560 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.226790 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.227160 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.227163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.227164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.227284 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.227697 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:36.227705 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:36.227966 (+ 261us) client_negotiation.cc:769] Sending connection context
0929 01:02:36.228187 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 01:02:36.228427 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":18,"thread_start_us":134,"threads_started":1}
I20260929 01:02:36.230542 7229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:02:36 dist-test-slave-7zdl krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643756, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 01:02:36.238289 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.230984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40081 (local address 127.0.0.1:58342)
0929 01:02:36.231083 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:36.231097 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:36.231161 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:36.231631 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:36.231634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:36.231784 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:36.231859 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.231863 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.232755 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.232761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.233335 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.233342 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.233797 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.233800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.233899 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.234237 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:36.234247 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:36.235416 (+ 1169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:36.236891 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.236904 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.236906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.237085 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.237364 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.237368 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.237370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.237446 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.237811 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:36.237814 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:36.237913 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 01:02:36.238020 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 01:02:36.238138 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 01:02:36.238339 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.231113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58342 (local address 127.0.98.61:40081)
0929 01:02:36.231408 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.231412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.231426 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:02:36.231466 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.231469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.231516 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.231621 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:36.231976 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.232653 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.233457 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.233693 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.233799 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.235544 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:36.235548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:36.235550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:36.235575 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:36.236776 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.237187 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.237191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.237192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.237274 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.237536 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.237541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.237543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.237720 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:36.237821 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.238069 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.238187 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":93,"threads_started":1}
I20260929 01:02:36.239147 7229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36721 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:02:36 dist-test-slave-7zdl krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643756, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 01:02:36.245997 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.239604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36721 (local address 127.0.0.1:60050)
0929 01:02:36.239720 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:36.239733 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:36.239809 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:36.240053 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:36.240055 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:36.240167 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:36.240228 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.240232 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.240937 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.240940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.241477 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.241484 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.241858 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.241861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.241915 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.242241 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:36.242251 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:36.243385 (+ 1134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:36.244930 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.244932 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.244933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.245092 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.245327 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.245329 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.245330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.245400 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.245680 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:36.245682 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:36.245744 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 01:02:36.245817 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 01:02:36.245900 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 01:02:36.246073 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.239664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60050 (local address 127.0.98.60:36721)
0929 01:02:36.239734 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.239740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.239775 (+ 35us) server_negotiation.cc:407] Connection header received
0929 01:02:36.239931 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.239933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.239972 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.240035 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:36.240336 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.240853 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.241562 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.241774 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.241845 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.243499 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:36.243501 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:36.243503 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:36.243522 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:36.244847 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.245189 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.245191 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.245192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.245255 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.245470 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.245472 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.245473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.245609 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:36.245678 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.245866 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.245956 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 01:02:36.249493 7229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036: Bootstrap starting.
I20260929 01:02:36.252102 7229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:36.252758 7229 log.cc:824] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:36.254465 7229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036: No bootstrap required, opened a new log
I20260929 01:02:36.257325 7229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36721 } }
I20260929 01:02:36.257511 7229 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:36.257550 7229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a6f4f441458470689115fea13835036, State: Initialized, Role: FOLLOWER
I20260929 01:02:36.257921 7229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36721 } }
I20260929 01:02:36.258306 7235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36721 } } }
I20260929 01:02:36.258451 7235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:02:36.259040 7229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:02:36.263923 7246 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:02:36.264026 7246 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:02:36.266873 7075 master_runner.cc:428] Detected that this master 1a6f4f441458470689115fea13835036 is joining an existing cluster
I20260929 01:02:36.266933 7075 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:36721
I20260929 01:02:36.270176 6702 heartbeater.cc:503] Master 127.0.98.62:44771 was elected leader, sending a full tablet report...
I20260929 01:02:36.278149 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.269301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33522 (local address 127.0.98.62:44771)
0929 01:02:36.269427 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.269432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.269724 (+ 292us) server_negotiation.cc:407] Connection header received
0929 01:02:36.269870 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.269873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.269930 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.270018 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:36.270912 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.271641 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.272910 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.273232 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.273319 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.274651 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:36.274655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:36.274658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:36.274681 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:36.276276 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.276851 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.276858 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.276863 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.276966 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.277263 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.277267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.277269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.277480 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:36.277550 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.277882 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.278022 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 01:02:36.278142 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.269144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.0.1:33522)
0929 01:02:36.269626 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:36.269641 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:36.269744 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:36.270247 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:36.270250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:36.270576 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:36.270752 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.270762 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.271759 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.271762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.272794 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.272803 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.273492 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.273495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.273587 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.274115 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:36.274131 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:36.274523 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:36.276401 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.276404 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.276406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.276726 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.277058 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.277061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.277063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.277153 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.277618 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:36.277621 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:36.277700 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 01:02:36.277803 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 01:02:36.277936 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":74,"thread_start_us":98,"threads_started":1}
I20260929 01:02:36.280293 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.269864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60054 (local address 127.0.98.60:36721)
0929 01:02:36.270162 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.270166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.271839 (+ 1673us) server_negotiation.cc:407] Connection header received
0929 01:02:36.271980 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.271982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.272018 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.272060 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:36.272531 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.273242 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.274259 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.274588 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.274683 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.275970 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:36.275975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:36.275977 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:36.275997 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:36.277508 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.278400 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.278407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.278410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.278501 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.279500 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.279506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.279508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.279666 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:36.279768 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.280037 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.280174 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":237}
I20260929 01:02:36.280303 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.269593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36721 (local address 127.0.0.1:60054)
0929 01:02:36.271766 (+ 2173us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:36.271779 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:36.271885 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:36.272159 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:36.272162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:36.272338 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:36.272430 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.272435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.273348 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.273351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.274141 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.274151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.274778 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.274782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.274892 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.275427 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:36.275441 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:36.275837 (+ 396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:36.277631 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.277635 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.277637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.277918 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.279267 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.279270 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.279271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.279402 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.279757 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:36.279760 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:36.279843 (+ 83us) client_negotiation.cc:769] Sending connection context
0929 01:02:36.279965 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 01:02:36.280109 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2110,"thread_start_us":87,"threads_started":1}
I20260929 01:02:36.282565 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 6573
Sep 29 01:02:36 dist-test-slave-7zdl krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643754, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 01:02:36.289227 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36813
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44771
--webserver_port=36787
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:44771,127.0.98.61:40081
--tserver_master_addrs=127.0.98.62:44771,127.0.98.61:40081,127.0.98.60:36721 with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:02:36.294610 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.282115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36721 (local address 127.0.0.1:60068)
0929 01:02:36.282241 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:36.282257 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:36.282330 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:36.283192 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:36.283194 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:36.283368 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:36.283456 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.283461 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.284157 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.284161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.284699 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.284706 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.285152 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.285155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.285242 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.285666 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:36.285677 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:36.287075 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:36.292836 (+ 5761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.292841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.292845 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.293136 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.293476 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.293479 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.293482 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.293576 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.294054 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:36.294059 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:36.294162 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 01:02:36.294265 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 01:02:36.294423 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 01:02:36.295311 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.282315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60068 (local address 127.0.98.60:36721)
0929 01:02:36.282514 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.282517 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.282527 (+ 10us) server_negotiation.cc:407] Connection header received
0929 01:02:36.282576 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.282578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.282613 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.282709 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:36.283566 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.284049 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.284813 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.285045 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.285151 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.287195 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:36.287198 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:36.287201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:36.287223 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:36.292675 (+ 5452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.293286 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.293289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.293291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.293368 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.293681 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.293686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.293691 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.293910 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:36.294007 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.295047 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.295157 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260929 01:02:36.295719 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.269428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58356 (local address 127.0.98.61:40081)
0929 01:02:36.269540 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.269544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.269940 (+ 396us) server_negotiation.cc:407] Connection header received
0929 01:02:36.270087 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.270090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.270143 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.270193 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:36.270943 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.271734 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.272934 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.273286 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.273396 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.291917 (+ 18521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:36.291921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:36.291924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:36.291982 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:36.293772 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.294250 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.294256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.294260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.294337 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.294631 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.294634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.294636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.294877 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:36.294962 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.295403 (+ 441us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.295564 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 01:02:36.296052 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.269144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40081 (local address 127.0.0.1:58356)
0929 01:02:36.269862 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:36.269875 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:36.269970 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:36.270329 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:36.270331 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:36.270627 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:36.270778 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.270785 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.271862 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.271865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.272805 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.272815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.273537 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.273542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.273664 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.291258 (+ 17594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:36.291277 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:36.291722 (+ 445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:36.293883 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.293886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.293888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.294119 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.294436 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.294439 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.294441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.294532 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.295005 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:36.295008 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:36.295086 (+ 78us) client_negotiation.cc:769] Sending connection context
0929 01:02:36.295184 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 01:02:36.295299 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":73,"spinlock_wait_cycles":8576,"thread_start_us":90,"threads_started":1}
I20260929 01:02:36.297097 6511 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:36721: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1a6f4f441458470689115fea13835036"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 36721
}
attrs {
promote: true
}
}
dest_uuid: "1d621ac6ce2e468f8451220161009672"
cas_config_opid_index: 5
I20260929 01:02:36.297787 6511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: true } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: true } }
I20260929 01:02:36.298523 7235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:02:36.298699 7235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36721 } }
I20260929 01:02:36.299640 7235 leader_election.cc:288] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d621ac6ce2e468f8451220161009672 (127.0.98.62:44771), 7bcff1ee79b34e0cb34c599ba53620e9 (127.0.98.61:40081)
I20260929 01:02:36.299964 6521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a6f4f441458470689115fea13835036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d621ac6ce2e468f8451220161009672" is_pre_election: true
I20260929 01:02:36.300604 7121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a6f4f441458470689115fea13835036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" is_pre_election: true
I20260929 01:02:36.300793 7121 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [term 1 LEARNER]: Handling vote request from an unknown peer 1a6f4f441458470689115fea13835036
I20260929 01:02:36.300951 7121 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 1a6f4f441458470689115fea13835036 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260929 01:02:36.301321 7178 leader_election.cc:302] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1a6f4f441458470689115fea13835036; no voters: 1d621ac6ce2e468f8451220161009672, 7bcff1ee79b34e0cb34c599ba53620e9
I20260929 01:02:36.302063 7071 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1a6f4f441458470689115fea13835036 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: true } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: true } } }
I20260929 01:02:36.303539 7071 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:36721
I20260929 01:02:36.303699 7071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1d621ac6ce2e468f8451220161009672" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: true } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: true } } }
I20260929 01:02:36.303814 7071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:36.304905 7203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:36.306284 7203 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [term 1 FOLLOWER]: Refusing update from remote peer 1d621ac6ce2e468f8451220161009672: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:02:36.306591 7075 master.cc:537] Master@127.0.98.60:36721 shutting down...
I20260929 01:02:36.308468 7070 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:36.308655 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.302173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40081 (local address 127.0.0.1:58368)
0929 01:02:36.302469 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:36.302484 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:36.302562 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:36.304476 (+ 1914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:36.304479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:36.304496 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:36.304588 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.304594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.305654 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.305658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.307059 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.307072 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.308219 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.308225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.308321 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.308363 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 01:02:36.308455 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 01:02:36.308531 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212}
I20260929 01:02:36.309247 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.304022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58368 (local address 127.0.98.61:40081)
0929 01:02:36.304128 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.304132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.304145 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:02:36.304185 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.304191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.304257 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.304345 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:36.304713 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.305519 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.307219 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.307844 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.307967 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.308156 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.309058 (+ 902us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.309130 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 01:02:36.309903 7121 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [term 1 LEARNER]: Refusing update from remote peer 1d621ac6ce2e468f8451220161009672: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260929 01:02:36.310518 6495 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 -> Peer 1a6f4f441458470689115fea13835036 (127.0.98.60:36721): Couldn't send request to peer 1a6f4f441458470689115fea13835036. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 01:02:36.310896 7070 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:02:36.312408 7235 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 01:02:36.312783 7121 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1a6f4f441458470689115fea13835036 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: true } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: true } } }
I20260929 01:02:36.313947 7075 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:36.314195 7075 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:36.314306 7075 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036: stopping tablet replica
I20260929 01:02:36.315475 7150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d621ac6ce2e468f8451220161009672. Latest consensus state: current_term: 1 leader_uuid: "1d621ac6ce2e468f8451220161009672" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: true } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: true } } }
I20260929 01:02:36.315589 7150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [sys.catalog]: This master's current role is: LEARNER
I20260929 01:02:36.315686 7150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d621ac6ce2e468f8451220161009672" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: true } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: true } } }
I20260929 01:02:36.315780 7150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [sys.catalog]: This master's current role is: LEARNER
I20260929 01:02:36.319207 7256 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672: attempting to promote NON_VOTER 7bcff1ee79b34e0cb34c599ba53620e9 to VOTER
I20260929 01:02:36.319625 7256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: true } }
I20260929 01:02:36.320791 7121 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [term 1 LEARNER]: Refusing update from remote peer 1d621ac6ce2e468f8451220161009672: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 01:02:36.320866 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.320595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36721 (local address 127.0.0.1:60070)
0929 01:02:36.320726 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:36.320790 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 01:02:36.321219 7256 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 01:02:36.321589 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.321351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36721 (local address 127.0.0.1:60080)
0929 01:02:36.321477 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:36.321527 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 01:02:36.321697 6495 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 -> Peer 1a6f4f441458470689115fea13835036 (127.0.98.60:36721): Couldn't send request to peer 1a6f4f441458470689115fea13835036. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 01:02:36.322762 7070 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7bcff1ee79b34e0cb34c599ba53620e9 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: true } } }
I20260929 01:02:36.324023 7070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1d621ac6ce2e468f8451220161009672" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: true } } }
I20260929 01:02:36.324131 7070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:36.324262 7070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d621ac6ce2e468f8451220161009672" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: true } } }
I20260929 01:02:36.324353 7070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:36.324852 7120 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7bcff1ee79b34e0cb34c599ba53620e9 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: true } } }
I20260929 01:02:36.326105 7258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d621ac6ce2e468f8451220161009672" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: true } } }
I20260929 01:02:36.326213 7258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:02:36.329489 7075 master.cc:552] Master@127.0.98.60:36721 shutdown complete.
I20260929 01:02:36.329556 7075 master_runner.cc:305] Clearing existing system tablet
I20260929 01:02:36.329782 7075 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:36.329849 7075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:36.330958 7075 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 01:02:36.331791 7260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:36.331983 7075 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:36.332106 7075 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:36.332142 7075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/wal
uuid: "1a6f4f441458470689115fea13835036"
format_stamp: "Formatted at 2026-09-29 01:02:36 on dist-test-slave-7zdl"
server_key: "65041022dffa16cce8166651bc14ec70"
server_key_iv: "657fd4d697bba53051aa8c685712880a"
server_key_version: "encryptionkey@0"
I20260929 01:02:36.333793 7075 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:36.335796 7075 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 01:02:36.335892 7075 log.cc:1196] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 01:02:36.336114 7075 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036: Deleting consensus metadata
I20260929 01:02:36.336292 7075 master_runner.cc:315] Copying system tablet from 127.0.98.62:44771
I20260929 01:02:36.337339 7075 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:44771
I20260929 01:02:36.345485 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.337924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33536 (local address 127.0.98.62:44771)
0929 01:02:36.338055 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.338059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.338264 (+ 205us) server_negotiation.cc:407] Connection header received
0929 01:02:36.338400 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.338403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.338480 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.338573 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:36.339170 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.339772 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.340731 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.341040 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.341136 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.342279 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:36.342283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:36.342286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:36.342309 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:36.343827 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.344299 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.344304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.344306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.344386 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.344670 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.344676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.344677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.344847 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:36.344949 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.345217 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.345317 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 01:02:36.345500 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.337881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.0.1:33536)
0929 01:02:36.338173 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:36.338184 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:36.338308 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:36.338591 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:36.338594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:36.338867 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:36.339012 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.339020 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.339880 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.339886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.340608 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.340618 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.341139 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.341143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.341240 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.341807 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:36.341828 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:36.342144 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:36.344008 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.344015 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.344017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.344199 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.344494 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.344498 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.344500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.344584 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.344957 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:36.344963 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:36.345064 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 01:02:36.345167 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 01:02:36.345306 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":98,"threads_started":1}
I20260929 01:02:36.346238 6531 tablet_copy_service.cc:140] P 1d621ac6ce2e468f8451220161009672: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1a6f4f441458470689115fea13835036 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33536)
I20260929 01:02:36.346349 6531 tablet_copy_service.cc:161] P 1d621ac6ce2e468f8451220161009672: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1a6f4f441458470689115fea13835036 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33536: session id = 1a6f4f441458470689115fea13835036-00000000000000000000000000000000
I20260929 01:02:36.347244 6531 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 01:02:36.348284 7075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 01:02:36.349902 7075 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036: tablet copy: Starting download of 0 data blocks...
I20260929 01:02:36.350096 7075 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036: tablet copy: Starting download of 1 WAL segments...
I20260929 01:02:36.351384 7075 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 01:02:36.352771 6531 tablet_copy_service.cc:342] P 1d621ac6ce2e468f8451220161009672: Request end of tablet copy session 1a6f4f441458470689115fea13835036-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33536
I20260929 01:02:36.352864 6531 tablet_copy_service.cc:433] P 1d621ac6ce2e468f8451220161009672: ending tablet copy session 1a6f4f441458470689115fea13835036-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1a6f4f441458470689115fea13835036
I20260929 01:02:36.359372 7075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:36.361101 7270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:36.361438 7268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:36.361562 7267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:36.361689 7075 server_base.cc:1034] running on GCE node
I20260929 01:02:36.361825 7075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:36.362139 7075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:36.363270 7075 hybrid_clock.cc:648] HybridClock initialized: now 1790643756363264 us; error 24 us; skew 500 ppm
Sep 29 01:02:36 dist-test-slave-7zdl krb5kdc[6466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643756, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:36.364728 7075 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260929 01:02:36.365180 7075 webserver.cc:460] Webserver started at http://127.0.98.60:44697/ using document root <none> and password file <none>
I20260929 01:02:36.365358 7075 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:36.365411 7075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:36.366405 7075 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:36.367446 7277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:36.367686 7075 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 01:02:36.367790 7075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/wal
uuid: "1a6f4f441458470689115fea13835036"
format_stamp: "Formatted at 2026-09-29 01:02:36 on dist-test-slave-7zdl"
server_key: "65041022dffa16cce8166651bc14ec70"
server_key_iv: "657fd4d697bba53051aa8c685712880a"
server_key_version: "encryptionkey@0"
I20260929 01:02:36.367887 7075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:36.395416 7075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:36.397390 7075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:36.402300 7075 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36721
I20260929 01:02:36.402915 7075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/master-2/data/info.pb
I20260929 01:02:36.402925 7329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36721 every 8 connection(s)
I20260929 01:02:36.404356 7330 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:02:36.405277 7330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036: Bootstrap starting.
I20260929 01:02:36.413642 7330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:02:36.413947 7330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036: Bootstrap complete.
I20260929 01:02:36.414433 7330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: true } }
I20260929 01:02:36.414583 7330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1a6f4f441458470689115fea13835036, State: Initialized, Role: LEARNER
I20260929 01:02:36.414685 7330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: true } }
I20260929 01:02:36.415035 7333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: true } } }
I20260929 01:02:36.415144 7330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:02:36.415216 7333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [sys.catalog]: This master's current role is: LEARNER
I20260929 01:02:36.418253 7344 catalog_manager.cc:1259] Loaded cluster ID: 3792c0e2458b4d2088777c90f9a31006
I20260929 01:02:36.418358 7344 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036: loading cluster ID for follower catalog manager: success
I20260929 01:02:36.420007 7344 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036: acquiring CA information for follower catalog manager: success
I20260929 01:02:36.421062 7344 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:02:36.439986 7255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:36.440320 7255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:36.440400 7255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:36.446147 7255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:36.446214 7255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:36.446296 7255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:02:36.453554 7255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36787
--tserver_master_addrs=127.0.98.62:44771,127.0.98.61:40081,127.0.98.60:36721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:36.454846 7255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:36.456135 7255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:36.467625 7349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:36.467660 7350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:36.467625 7352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:36.467949 7255 server_base.cc:1034] running on GCE node
I20260929 01:02:36.468400 7255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:36.469071 7255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:36.470247 7255 hybrid_clock.cc:648] HybridClock initialized: now 1790643756470217 us; error 39 us; skew 500 ppm
Sep 29 01:02:36 dist-test-slave-7zdl krb5kdc[6466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643756, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:36.473376 7255 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 01:02:36.474234 7255 webserver.cc:460] Webserver started at http://127.0.98.1:36787/ using document root <none> and password file <none>
I20260929 01:02:36.474833 7255 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:36.474915 7255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:36.478894 7255 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:02:36.480391 6836 heartbeater.cc:503] Master 127.0.98.62:44771 was elected leader, sending a full tablet report...
I20260929 01:02:36.481282 7359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:36.482560 7255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:02:36.482649 7255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23"
format_stamp: "Formatted at 2026-09-29 01:02:35 on dist-test-slave-7zdl"
server_key: "15433554939ea8ee96b3e8d8229076ec"
server_key_iv: "eeec67d802311488abbfd6308d58226d"
server_key_version: "encryptionkey@0"
I20260929 01:02:36.483228 7255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:36.494432 7255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:36.497016 7255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:36.497153 7255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:36.497632 7255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:36.498384 7255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:36.498430 7255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:36.498466 7255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:36.498488 7255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:36.507431 7255 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36813
I20260929 01:02:36.507513 7472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36813 every 8 connection(s)
I20260929 01:02:36.508409 7255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:02:36.516330 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 7255
I20260929 01:02:36.516525 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 6707
Sep 29 01:02:36 dist-test-slave-7zdl krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643756, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 29 01:02:36 dist-test-slave-7zdl krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643756, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 01:02:36 dist-test-slave-7zdl krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643756, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 01:02:36.524462 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42313
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44771
--webserver_port=40497
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:44771,127.0.98.61:40081
--tserver_master_addrs=127.0.98.62:44771,127.0.98.61:40081,127.0.98.60:36721 with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:02:36.525720 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.510813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37301 (local address 127.0.98.61:40081)
0929 01:02:36.510924 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.510930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.511249 (+ 319us) server_negotiation.cc:407] Connection header received
0929 01:02:36.512350 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.512355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.512417 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.512484 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:36.513762 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.514562 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.516285 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.516611 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.516688 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.521374 (+ 4686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:36.521378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:36.521380 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:36.521410 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:36.523174 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.523837 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.523841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.523843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.523933 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.524369 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.524372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.524374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.524616 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:36.524703 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.525422 (+ 719us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.525540 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 01:02:36.526450 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.510116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39867 (local address 127.0.98.62:44771)
0929 01:02:36.510494 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.510499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.511739 (+ 1240us) server_negotiation.cc:407] Connection header received
0929 01:02:36.512325 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.512328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.512382 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.512490 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:36.513716 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.514562 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.516251 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.516611 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.516720 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.521127 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:36.521133 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:36.521140 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:36.521173 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:36.523892 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.524418 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.524421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.524423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.524508 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.525196 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.525201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.525206 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.525451 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:36.525543 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.526191 (+ 648us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.526300 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291}
I20260929 01:02:36.526757 7480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.510569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.98.1:39867)
0929 01:02:36.511651 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:36.511662 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:36.512131 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:36.512657 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:36.512664 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:36.513050 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:36.513579 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.513592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.514735 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.514739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.516102 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.516116 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.516900 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.516906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.517235 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.518043 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:36.518068 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:36.520903 (+ 2835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:36.524036 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.524041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.524044 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.524304 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.524967 (+ 663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.524973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.524978 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.525083 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.525597 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:36.525603 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:36.525705 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 01:02:36.526118 (+ 413us) client_negotiation.cc:240] Negotiation successful
0929 01:02:36.526256 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1004,"mutex_wait_us":164,"spinlock_wait_cycles":57728,"thread_start_us":716,"threads_started":1}
I20260929 01:02:36.527110 7478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.510569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40081 (local address 127.0.98.1:37301)
0929 01:02:36.511117 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:36.511155 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:36.512128 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:36.512659 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:36.512666 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:36.513107 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:36.513603 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.513614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.514901 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.514911 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.516103 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.516109 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.518571 (+ 2462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.518574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.518683 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.519268 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:36.519289 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:36.521220 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:36.523327 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.523334 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.523350 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.523720 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.524062 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.524064 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.524066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.524264 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.524766 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:36.524773 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:36.525082 (+ 309us) client_negotiation.cc:769] Sending connection context
0929 01:02:36.525956 (+ 874us) client_negotiation.cc:240] Negotiation successful
0929 01:02:36.526081 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":234,"spinlock_wait_cycles":46976,"thread_start_us":160,"threads_started":1}
I20260929 01:02:36.527292 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.510246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57309 (local address 127.0.98.60:36721)
0929 01:02:36.510633 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.510638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.511919 (+ 1281us) server_negotiation.cc:407] Connection header received
0929 01:02:36.512418 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.512421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.512475 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.512549 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:36.513744 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.514589 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.516249 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.516716 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.516804 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.521738 (+ 4934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:36.521741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:36.521744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:36.521773 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:36.523479 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.523950 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.523955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.523980 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.524066 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.524570 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.524574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.524576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.524788 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:36.524870 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.526915 (+ 2045us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.527088 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":85,"threads_started":1}
I20260929 01:02:36.526770 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.510570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36721 (local address 127.0.98.1:57309)
0929 01:02:36.511841 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:36.511850 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:36.512128 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:36.512659 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:36.512666 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:36.513112 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:36.513611 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.513617 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.514745 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.514748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.516098 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.516108 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.516919 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.516924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.517264 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.518165 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:36.518185 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:36.521614 (+ 3429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:36.523594 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.523597 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.523599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.523849 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.524374 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.524376 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.524379 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.524472 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.524918 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:36.524920 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:36.525082 (+ 162us) client_negotiation.cc:769] Sending connection context
0929 01:02:36.525331 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 01:02:36.525684 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1208,"mutex_wait_us":464,"spinlock_wait_cycles":1361536,"thread_start_us":730,"threads_started":1}
I20260929 01:02:36.528132 7474 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44771
I20260929 01:02:36.528350 7474 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:36.528523 7473 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36721
I20260929 01:02:36.528599 7473 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:36.529047 7474 heartbeater.cc:511] Master 127.0.98.62:44771 requested a full tablet report, sending...
I20260929 01:02:36.529047 7473 heartbeater.cc:511] Master 127.0.98.60:36721 requested a full tablet report, sending...
I20260929 01:02:36.529620 7475 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40081
I20260929 01:02:36.529690 7475 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:36.529804 7475 heartbeater.cc:511] Master 127.0.98.61:40081 requested a full tablet report, sending...
I20260929 01:02:36.530725 7111 ts_manager.cc:194] Registered new tserver with Master: fb1e8fd0fcac4b96bf8d5c75404d2e23 (127.0.98.1:36813)
I20260929 01:02:36.530723 7294 ts_manager.cc:194] Registered new tserver with Master: fb1e8fd0fcac4b96bf8d5c75404d2e23 (127.0.98.1:36813)
I20260929 01:02:36.530740 6511 ts_manager.cc:194] Re-registered known tserver with Master: fb1e8fd0fcac4b96bf8d5c75404d2e23 (127.0.98.1:36813)
I20260929 01:02:36.531677 6511 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39867
W20260929 01:02:36.638001 7481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:36.638312 7481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:36.638386 7481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:36.642761 7481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:36.642869 7481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:36.642988 7481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:02:36.648301 7481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40497
--tserver_master_addrs=127.0.98.62:44771,127.0.98.61:40081,127.0.98.60:36721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:36.649402 7481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:36.650343 7481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:36.659381 7489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:36.659396 7486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:36.659399 7487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:36.659799 7481 server_base.cc:1034] running on GCE node
I20260929 01:02:36.660192 7481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:36.660796 7481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:36.661952 7481 hybrid_clock.cc:648] HybridClock initialized: now 1790643756661917 us; error 42 us; skew 500 ppm
Sep 29 01:02:36 dist-test-slave-7zdl krb5kdc[6466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643756, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:36.665611 7481 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 01:02:36.666488 7481 webserver.cc:460] Webserver started at http://127.0.98.2:40497/ using document root <none> and password file <none>
I20260929 01:02:36.667100 7481 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:36.667191 7481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:36.671000 7481 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 01:02:36.673074 7496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:36.674431 7481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:36.674572 7481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "d158538b2a3d427182b72f8694c1e589"
format_stamp: "Formatted at 2026-09-29 01:02:35 on dist-test-slave-7zdl"
server_key: "12cb589c1a78882abdbf8c9a1eb97dd8"
server_key_iv: "4c23d6aba532bad0f33c89aeec21902a"
server_key_version: "encryptionkey@0"
I20260929 01:02:36.675293 7481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:36.691490 7481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:36.694578 7481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:36.694720 7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:36.695267 7481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:36.696116 7481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:36.696161 7481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:36.696198 7481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:36.696218 7481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:36.699128 6970 heartbeater.cc:503] Master 127.0.98.62:44771 was elected leader, sending a full tablet report...
I20260929 01:02:36.705848 7481 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42313
I20260929 01:02:36.705868 7609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42313 every 8 connection(s)
I20260929 01:02:36.706857 7481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:02:36.712980 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 7481
I20260929 01:02:36.713157 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 6841
Sep 29 01:02:36 dist-test-slave-7zdl krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643756, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 01:02:36 dist-test-slave-7zdl krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643756, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 29 01:02:36 dist-test-slave-7zdl krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643756, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 01:02:36.721460 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34461
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44771
--webserver_port=42459
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:44771,127.0.98.61:40081
--tserver_master_addrs=127.0.98.62:44771,127.0.98.61:40081,127.0.98.60:36721 with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:02:36.725224 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.708781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42447 (local address 127.0.98.62:44771)
0929 01:02:36.708919 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.708923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.709845 (+ 922us) server_negotiation.cc:407] Connection header received
0929 01:02:36.711098 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.711101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.711156 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.711228 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:36.712519 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.713277 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.715764 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.716239 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.716331 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.720964 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:36.720971 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:36.720977 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:36.721011 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:36.723148 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.723667 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.723671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.723673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.723748 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.724074 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.724077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.724079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.724291 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:36.724365 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.724932 (+ 567us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.725052 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 01:02:36.725391 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.708592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51129 (local address 127.0.98.61:40081)
0929 01:02:36.708732 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.708736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.710062 (+ 1326us) server_negotiation.cc:407] Connection header received
0929 01:02:36.711112 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.711116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.711168 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.711220 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:36.712538 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.713339 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.715695 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.716055 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.716137 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.721325 (+ 5188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:36.721328 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:36.721331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:36.721359 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:36.723389 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.723902 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.723906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.723909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.724015 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.724325 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.724330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.724333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.724586 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:36.724674 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.725030 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.725166 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 01:02:36.726310 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.709096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.98.2:42447)
0929 01:02:36.709707 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:36.709756 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:36.710900 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:36.711395 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:36.711402 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:36.711844 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:36.712379 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.712393 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.713964 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.713971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.715566 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.715579 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.716541 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.716547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.716699 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.717736 (+ 1037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:36.717756 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:36.720809 (+ 3053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:36.723319 (+ 2510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.723322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.723324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.723553 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.723870 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.723873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.723875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.723954 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.724415 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:36.724417 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:36.724626 (+ 209us) client_negotiation.cc:769] Sending connection context
0929 01:02:36.725375 (+ 749us) client_negotiation.cc:240] Negotiation successful
0929 01:02:36.725501 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":193,"spinlock_wait_cycles":4608,"thread_start_us":137,"threads_started":1}
I20260929 01:02:36.726576 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.709097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40081 (local address 127.0.98.2:51129)
0929 01:02:36.709981 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:36.709993 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:36.710893 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:36.711397 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:36.711403 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:36.711889 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:36.712380 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.712394 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.713795 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.713802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.715543 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.715549 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.716351 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.716355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.716689 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.717556 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:36.717579 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:36.721159 (+ 3580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:36.723513 (+ 2354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.723518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.723520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.723789 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.724131 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.724136 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.724139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.724222 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.724733 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:36.724739 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:36.724844 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 01:02:36.725647 (+ 803us) client_negotiation.cc:240] Negotiation successful
0929 01:02:36.725813 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":398,"spinlock_wait_cycles":9728,"thread_start_us":112,"threads_started":1}
I20260929 01:02:36.726732 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.709289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42203 (local address 127.0.98.60:36721)
0929 01:02:36.709425 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.709429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.710316 (+ 887us) server_negotiation.cc:407] Connection header received
0929 01:02:36.711113 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.711116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.711169 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.711220 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:36.712539 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.713340 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.715707 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.716069 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.716137 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.720436 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:36.720439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:36.720441 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:36.720492 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:36.722607 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.723296 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.723308 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.723309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.723404 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.723878 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.723882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.723884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.724136 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:36.724224 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.726429 (+ 2205us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.726580 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 01:02:36.726294 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.709196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36721 (local address 127.0.98.2:42203)
0929 01:02:36.710207 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:36.710231 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:36.710893 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:36.711397 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:36.711403 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:36.711884 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:36.712382 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.712394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.714069 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.714072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.715533 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.715545 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.716391 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.716397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.716716 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.717595 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:36.717606 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:36.720254 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:36.722742 (+ 2488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.722749 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.722767 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.723141 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.723518 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.723522 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.723524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.723769 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.724274 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:36.724282 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:36.724627 (+ 345us) client_negotiation.cc:769] Sending connection context
0929 01:02:36.724857 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 01:02:36.725201 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"mutex_wait_us":515,"spinlock_wait_cycles":99968,"thread_start_us":124,"threads_started":1}
I20260929 01:02:36.727682 7612 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40081
I20260929 01:02:36.727725 7611 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44771
I20260929 01:02:36.727931 7611 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:36.727931 7612 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:36.728530 7611 heartbeater.cc:511] Master 127.0.98.62:44771 requested a full tablet report, sending...
I20260929 01:02:36.728528 7612 heartbeater.cc:511] Master 127.0.98.61:40081 requested a full tablet report, sending...
I20260929 01:02:36.729733 6511 ts_manager.cc:194] Re-registered known tserver with Master: d158538b2a3d427182b72f8694c1e589 (127.0.98.2:42313)
I20260929 01:02:36.729988 7111 ts_manager.cc:194] Registered new tserver with Master: d158538b2a3d427182b72f8694c1e589 (127.0.98.2:42313)
I20260929 01:02:36.730630 6511 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42447
I20260929 01:02:36.730849 7610 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36721
I20260929 01:02:36.730953 7610 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:36.731130 7610 heartbeater.cc:511] Master 127.0.98.60:36721 requested a full tablet report, sending...
I20260929 01:02:36.731946 7294 ts_manager.cc:194] Registered new tserver with Master: d158538b2a3d427182b72f8694c1e589 (127.0.98.2:42313)
W20260929 01:02:36.834903 7617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:36.835140 7617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:36.835203 7617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:36.839179 7617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:36.839255 7617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:36.839337 7617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:02:36.844511 7617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42459
--tserver_master_addrs=127.0.98.62:44771,127.0.98.61:40081,127.0.98.60:36721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:36.845606 7617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:36.846606 7617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:36.855824 7625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:36.855953 7622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:36.856016 7617 server_base.cc:1034] running on GCE node
W20260929 01:02:36.856047 7623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:36.856379 7617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:36.856969 7617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:36.858119 7617 hybrid_clock.cc:648] HybridClock initialized: now 1790643756858107 us; error 24 us; skew 500 ppm
Sep 29 01:02:36 dist-test-slave-7zdl krb5kdc[6466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643756, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:36.860961 7617 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 01:02:36.861850 7617 webserver.cc:460] Webserver started at http://127.0.98.3:42459/ using document root <none> and password file <none>
I20260929 01:02:36.862445 7617 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:36.862517 7617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:36.866513 7617 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:02:36.868005 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.864019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60094 (local address 127.0.98.60:36721)
0929 01:02:36.864146 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.864152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.864242 (+ 90us) server_negotiation.cc:407] Connection header received
0929 01:02:36.864412 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.864415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.864473 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.864569 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:36.864822 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.865330 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.866528 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.867287 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.867395 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.867552 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.867800 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.867863 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:02:36.868009 7631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.863831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36721 (local address 127.0.0.1:60094)
0929 01:02:36.864146 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:36.864161 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:36.864304 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:36.864583 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:36.864586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:36.864605 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:36.864717 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.864723 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.865441 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.865446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.866406 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.866417 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.867606 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.867609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.867678 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.867696 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 01:02:36.867788 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 01:02:36.867864 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":66,"threads_started":1}
I20260929 01:02:36.868757 7304 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 01:02:36.868942 7633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:36.869163 7333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d621ac6ce2e468f8451220161009672. Latest consensus state: current_term: 1 leader_uuid: "1d621ac6ce2e468f8451220161009672" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: true } } }
I20260929 01:02:36.869315 7333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [sys.catalog]: This master's current role is: LEARNER
I20260929 01:02:36.870306 7617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 01:02:36.870428 7617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "1555431512ad4290b1cd00b2a315816e"
format_stamp: "Formatted at 2026-09-29 01:02:35 on dist-test-slave-7zdl"
server_key: "fece09d6e1a850524f7b3a279c3308f4"
server_key_iv: "fa1fef084ab3490021a40a6e868693db"
server_key_version: "encryptionkey@0"
I20260929 01:02:36.871016 7617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:36.889319 7617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:36.892370 7617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:36.892623 7617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:36.893244 7617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:36.894100 7617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:36.894174 7617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:36.894244 7617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:36.894287 7617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:36.906678 7617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34461
I20260929 01:02:36.906730 7746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34461 every 8 connection(s)
I20260929 01:02:36.907835 7617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:02:36.913379 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 7617
Sep 29 01:02:36 dist-test-slave-7zdl krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643756, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 29 01:02:36 dist-test-slave-7zdl krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643756, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 01:02:36 dist-test-slave-7zdl krb5kdc[6466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643756, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 01:02:36.925431 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.909731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37451 (local address 127.0.98.62:44771)
0929 01:02:36.909897 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.909902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.911239 (+ 1337us) server_negotiation.cc:407] Connection header received
0929 01:02:36.912365 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.912369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.912427 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.912514 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:36.914364 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.915203 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.916741 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.917098 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.917180 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.921570 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:36.921576 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:36.921580 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:36.921619 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:36.923505 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.924028 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.924032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.924034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.924123 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.924457 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.924461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.924462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.924666 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:36.924742 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.925137 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.925257 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 01:02:36.926167 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.911724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36721 (local address 127.0.98.3:43253)
0929 01:02:36.912534 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:36.912547 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:36.912634 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:36.912876 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:36.912879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:36.913337 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:36.914171 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.914193 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.915257 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.915259 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.916447 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.916462 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.917227 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.917231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.917457 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.918426 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:36.918441 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:36.920284 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:36.922683 (+ 2399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.922691 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.922726 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.923120 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.923484 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.923488 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.923490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.923661 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.924212 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:36.924220 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:36.924526 (+ 306us) client_negotiation.cc:769] Sending connection context
0929 01:02:36.924749 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 01:02:36.925083 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":241,"spinlock_wait_cycles":8960,"thread_start_us":130,"threads_started":1}
I20260929 01:02:36.926350 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.910130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.98.3:37451)
0929 01:02:36.911081 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:36.911137 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:36.912147 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:36.912641 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:36.912650 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:36.913337 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:36.914172 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.914193 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.915375 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.915378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.916583 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.916595 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.917373 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.917377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.917541 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.918255 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:36.918305 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:36.921397 (+ 3092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:36.923636 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.923639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.923641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.923890 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.924251 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.924254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.924256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.924341 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.924816 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:36.924819 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:36.924909 (+ 90us) client_negotiation.cc:769] Sending connection context
0929 01:02:36.925561 (+ 652us) client_negotiation.cc:240] Negotiation successful
0929 01:02:36.925690 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":335,"spinlock_wait_cycles":17664,"thread_start_us":555,"threads_started":1}
I20260929 01:02:36.927016 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.911826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43253 (local address 127.0.98.60:36721)
0929 01:02:36.911960 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.911965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.912604 (+ 639us) server_negotiation.cc:407] Connection header received
0929 01:02:36.912709 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.912712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.912765 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.912851 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:36.914365 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.915111 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.916560 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.916923 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.917023 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.920508 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:36.920515 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:36.920519 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:36.920549 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:36.922534 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.923265 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.923271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.923274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.923366 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.923793 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.923798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.923800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.924055 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:36.924155 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.926681 (+ 2526us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.926844 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 01:02:36.927214 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.911195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47553 (local address 127.0.98.61:40081)
0929 01:02:36.911316 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.911320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.912102 (+ 782us) server_negotiation.cc:407] Connection header received
0929 01:02:36.912328 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.912331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.912385 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.912474 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:36.914364 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.915203 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.916742 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.917098 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.917180 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.922829 (+ 5649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:36.922833 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:36.922836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:36.922859 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:36.924589 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.925064 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.925067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.925069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.925154 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.925436 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.925439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.925441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.925629 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:36.925696 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.926961 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.927067 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 01:02:36.927407 7748 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44771
I20260929 01:02:36.927561 7747 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36721
I20260929 01:02:36.927644 7748 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:36.927644 7747 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:36.928303 7747 heartbeater.cc:511] Master 127.0.98.60:36721 requested a full tablet report, sending...
I20260929 01:02:36.928303 7748 heartbeater.cc:511] Master 127.0.98.62:44771 requested a full tablet report, sending...
I20260929 01:02:36.929013 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.911092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40081 (local address 127.0.98.3:47553)
0929 01:02:36.912004 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:36.912017 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:36.912146 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:36.912641 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:36.912650 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:36.913339 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:36.914176 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.914193 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.915375 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.915378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.916611 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.916619 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.917536 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.917540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.917618 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.918587 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:36.918604 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:36.921732 (+ 3128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:36.924709 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.924712 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.924714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.924952 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.925265 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:36.925267 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:36.925269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:36.925347 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:36.926060 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:36.926063 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:36.926157 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 01:02:36.928443 (+ 2286us) client_negotiation.cc:240] Negotiation successful
0929 01:02:36.928587 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":315,"thread_start_us":110,"threads_started":1}
I20260929 01:02:36.929554 7294 ts_manager.cc:194] Registered new tserver with Master: 1555431512ad4290b1cd00b2a315816e (127.0.98.3:34461)
I20260929 01:02:36.929877 7749 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40081
I20260929 01:02:36.929914 6511 ts_manager.cc:194] Re-registered known tserver with Master: 1555431512ad4290b1cd00b2a315816e (127.0.98.3:34461)
I20260929 01:02:36.930043 7749 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:36.930289 7749 heartbeater.cc:511] Master 127.0.98.61:40081 requested a full tablet report, sending...
I20260929 01:02:36.930698 6511 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37451
I20260929 01:02:36.931087 7111 ts_manager.cc:194] Registered new tserver with Master: 1555431512ad4290b1cd00b2a315816e (127.0.98.3:34461)
I20260929 01:02:36.935806 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.916833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60108 (local address 127.0.98.60:36721)
0929 01:02:36.920892 (+ 4059us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.920905 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.920918 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:02:36.920955 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.920958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.921008 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.921091 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:36.923629 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.924503 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.927830 (+ 3327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.928160 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.928251 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.931826 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:36.931830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:36.931833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:36.931870 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:36.933688 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.934283 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.934289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.934292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.934383 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.934743 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.934749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.934754 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.935031 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:36.935135 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.935449 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.935584 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3973,"thread_start_us":66,"threads_started":1}
I20260929 01:02:36.936983 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.917229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58376 (local address 127.0.98.61:40081)
0929 01:02:36.919524 (+ 2295us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.919527 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.920007 (+ 480us) server_negotiation.cc:407] Connection header received
0929 01:02:36.920137 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.920140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.920189 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.920305 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:36.921837 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.922651 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.929650 (+ 6999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.929956 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.930049 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.932835 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:36.932840 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:36.932844 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:36.932877 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:36.934797 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.935458 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.935461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.935463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.935544 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.935844 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.935847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.935850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.936354 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:36.936503 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.936712 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.936799 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2242,"thread_start_us":54,"threads_started":1}
I20260929 01:02:36.937003 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.917613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33544 (local address 127.0.98.62:44771)
0929 01:02:36.919406 (+ 1793us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.919410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.920495 (+ 1085us) server_negotiation.cc:407] Connection header received
0929 01:02:36.920536 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.920540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.920592 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.920673 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:36.922635 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.923498 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.926682 (+ 3184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.927627 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.927729 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.933189 (+ 5460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:36.933193 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:36.933196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:36.933218 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:36.934702 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.935264 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.935267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.935268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.935392 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:36.935756 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:36.935759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:36.935761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:36.935980 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:36.936108 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.936659 (+ 551us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.936797 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1707,"thread_start_us":81,"threads_started":1}
I20260929 01:02:36.939682 6511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:02:36.942530 6511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:02:36.944976 7766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d621ac6ce2e468f8451220161009672" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: true } } }
I20260929 01:02:36.945139 7766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:36.948284 7256 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672: attempting to promote NON_VOTER 1a6f4f441458470689115fea13835036 to VOTER
I20260929 01:02:36.948645 7256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: false } }
I20260929 01:02:36.949659 7120 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [term 1 FOLLOWER]: Refusing update from remote peer 1d621ac6ce2e468f8451220161009672: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 01:02:36.950030 7765 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 01:02:36.951298 7304 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [term 1 LEARNER]: Refusing update from remote peer 1d621ac6ce2e468f8451220161009672: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 01:02:36.951496 7765 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1a6f4f441458470689115fea13835036 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: false } } }
I20260929 01:02:36.952827 7765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1d621ac6ce2e468f8451220161009672" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: false } } }
I20260929 01:02:36.952803 7774 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 01:02:36.952920 7765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:36.953025 7765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d621ac6ce2e468f8451220161009672" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: false } } }
I20260929 01:02:36.953106 7765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:36.953620 7120 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1a6f4f441458470689115fea13835036 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: false } } }
I20260929 01:02:36.954160 7304 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1a6f4f441458470689115fea13835036 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: false } } }
I20260929 01:02:36.955365 7775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d621ac6ce2e468f8451220161009672" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: false } } }
I20260929 01:02:36.955487 7775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:02:36.956159 7333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d621ac6ce2e468f8451220161009672" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: false } } }
I20260929 01:02:36.956267 7333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:02:36.959210 7256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d621ac6ce2e468f8451220161009672" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: false } } }
I20260929 01:02:36.959321 7256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:36.963023 7631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.957498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42313 (local address 127.0.0.1:60872)
0929 01:02:36.957647 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:36.957660 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:36.957745 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:36.958766 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:36.958770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:36.958790 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:36.958921 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.958929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.960364 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.960367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.961431 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.961438 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.962606 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.962610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.962699 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.962724 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 01:02:36.962768 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 01:02:36.962893 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":52}
I20260929 01:02:36.963515 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.957911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60872 (local address 127.0.98.2:42313)
0929 01:02:36.958296 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.958306 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.958328 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:02:36.958412 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.958417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.958620 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.958773 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:36.959066 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.960248 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.961544 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.962473 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.963071 (+ 598us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.963221 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.963272 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.963364 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":95,"threads_started":1}
I20260929 01:02:36.965620 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.958037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36813 (local address 127.0.0.1:42936)
0929 01:02:36.959609 (+ 1572us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:36.959620 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:36.959686 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:36.960310 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:36.960313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:36.960333 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:36.961608 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.961613 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.962949 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.962955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.964060 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.964070 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.965246 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.965250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.965342 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.965363 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 01:02:36.965431 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 01:02:36.965497 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1522,"mutex_wait_us":5,"thread_start_us":97,"threads_started":1}
I20260929 01:02:36.965865 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.958026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34461 (local address 127.0.0.1:40964)
0929 01:02:36.959354 (+ 1328us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:36.959367 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:36.959449 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:36.961145 (+ 1696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:36.961148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:36.961166 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:36.961284 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.961291 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.962665 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.962668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.964299 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:36.964309 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.965488 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.965491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:36.965577 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.965597 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 01:02:36.965657 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 01:02:36.965737 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1268,"thread_start_us":104,"threads_started":1}
I20260929 01:02:36.966081 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.958417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42936 (local address 127.0.98.1:36813)
0929 01:02:36.959223 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.959234 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.959770 (+ 536us) server_negotiation.cc:407] Connection header received
0929 01:02:36.959849 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.959853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.960050 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.960179 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:36.961719 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.962819 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.964208 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.964960 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.965060 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.965185 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.965830 (+ 645us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.965935 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":126,"threads_started":1}
I20260929 01:02:36.966290 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:36.958947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40964 (local address 127.0.98.3:34461)
0929 01:02:36.960748 (+ 1801us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:36.960759 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:36.960780 (+ 21us) server_negotiation.cc:407] Connection header received
0929 01:02:36.960845 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:36.960850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:36.961012 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:36.961145 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:36.961409 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.962537 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.964444 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:36.965216 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:36.965310 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:36.965434 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:36.966030 (+ 596us) server_negotiation.cc:299] Negotiation successful
0929 01:02:36.966146 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1704,"thread_start_us":103,"threads_started":1}
I20260929 01:02:36.966684 7544 tablet_service.cc:1467] Processing CreateTablet for tablet 0e747d853a07402880d95b212251a93a (DEFAULT_TABLE table=test-table [id=af137da4d4c546b884c9b834a8d372cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:36.967833 7544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e747d853a07402880d95b212251a93a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:36.968773 7407 tablet_service.cc:1467] Processing CreateTablet for tablet 0e747d853a07402880d95b212251a93a (DEFAULT_TABLE table=test-table [id=af137da4d4c546b884c9b834a8d372cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:36.969135 7681 tablet_service.cc:1467] Processing CreateTablet for tablet 0e747d853a07402880d95b212251a93a (DEFAULT_TABLE table=test-table [id=af137da4d4c546b884c9b834a8d372cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:36.969722 7407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e747d853a07402880d95b212251a93a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:36.969950 7681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e747d853a07402880d95b212251a93a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:36.977816 7783 tablet_bootstrap.cc:492] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23: Bootstrap starting.
I20260929 01:02:36.979100 7782 tablet_bootstrap.cc:492] T 0e747d853a07402880d95b212251a93a P 1555431512ad4290b1cd00b2a315816e: Bootstrap starting.
I20260929 01:02:36.978755 7781 tablet_bootstrap.cc:492] T 0e747d853a07402880d95b212251a93a P d158538b2a3d427182b72f8694c1e589: Bootstrap starting.
I20260929 01:02:36.980105 7783 tablet_bootstrap.cc:654] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:36.980947 7783 log.cc:824] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:36.981922 7781 tablet_bootstrap.cc:654] T 0e747d853a07402880d95b212251a93a P d158538b2a3d427182b72f8694c1e589: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:36.982681 7781 log.cc:824] T 0e747d853a07402880d95b212251a93a P d158538b2a3d427182b72f8694c1e589: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:36.983002 7783 tablet_bootstrap.cc:492] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23: No bootstrap required, opened a new log
I20260929 01:02:36.983196 7783 ts_tablet_manager.cc:1397] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 01:02:36.984058 7782 tablet_bootstrap.cc:654] T 0e747d853a07402880d95b212251a93a P 1555431512ad4290b1cd00b2a315816e: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:36.984808 7782 log.cc:824] T 0e747d853a07402880d95b212251a93a P 1555431512ad4290b1cd00b2a315816e: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:36.986685 7782 tablet_bootstrap.cc:492] T 0e747d853a07402880d95b212251a93a P 1555431512ad4290b1cd00b2a315816e: No bootstrap required, opened a new log
I20260929 01:02:36.986847 7781 tablet_bootstrap.cc:492] T 0e747d853a07402880d95b212251a93a P d158538b2a3d427182b72f8694c1e589: No bootstrap required, opened a new log
I20260929 01:02:36.986967 7782 ts_tablet_manager.cc:1397] T 0e747d853a07402880d95b212251a93a P 1555431512ad4290b1cd00b2a315816e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 01:02:36.987043 7781 ts_tablet_manager.cc:1397] T 0e747d853a07402880d95b212251a93a P d158538b2a3d427182b72f8694c1e589: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20260929 01:02:36.987818 7783 raft_consensus.cc:348] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d158538b2a3d427182b72f8694c1e589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36813 } } peers { permanent_uuid: "1555431512ad4290b1cd00b2a315816e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34461 } }
I20260929 01:02:36.988334 7783 raft_consensus.cc:374] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:36.988411 7783 raft_consensus.cc:729] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e8fd0fcac4b96bf8d5c75404d2e23, State: Initialized, Role: FOLLOWER
I20260929 01:02:36.988888 7783 consensus_queue.cc:260] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d158538b2a3d427182b72f8694c1e589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36813 } } peers { permanent_uuid: "1555431512ad4290b1cd00b2a315816e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34461 } }
I20260929 01:02:36.989907 7783 ts_tablet_manager.cc:1428] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 01:02:36.990204 7474 heartbeater.cc:503] Master 127.0.98.62:44771 was elected leader, sending a full tablet report...
I20260929 01:02:36.991478 7781 raft_consensus.cc:348] T 0e747d853a07402880d95b212251a93a P d158538b2a3d427182b72f8694c1e589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d158538b2a3d427182b72f8694c1e589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36813 } } peers { permanent_uuid: "1555431512ad4290b1cd00b2a315816e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34461 } }
I20260929 01:02:36.991791 7781 raft_consensus.cc:374] T 0e747d853a07402880d95b212251a93a P d158538b2a3d427182b72f8694c1e589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:36.991847 7781 raft_consensus.cc:729] T 0e747d853a07402880d95b212251a93a P d158538b2a3d427182b72f8694c1e589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d158538b2a3d427182b72f8694c1e589, State: Initialized, Role: FOLLOWER
I20260929 01:02:36.992040 7782 raft_consensus.cc:348] T 0e747d853a07402880d95b212251a93a P 1555431512ad4290b1cd00b2a315816e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d158538b2a3d427182b72f8694c1e589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36813 } } peers { permanent_uuid: "1555431512ad4290b1cd00b2a315816e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34461 } }
I20260929 01:02:36.992306 7782 raft_consensus.cc:374] T 0e747d853a07402880d95b212251a93a P 1555431512ad4290b1cd00b2a315816e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:36.992362 7782 raft_consensus.cc:729] T 0e747d853a07402880d95b212251a93a P 1555431512ad4290b1cd00b2a315816e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1555431512ad4290b1cd00b2a315816e, State: Initialized, Role: FOLLOWER
I20260929 01:02:36.992314 7781 consensus_queue.cc:260] T 0e747d853a07402880d95b212251a93a P d158538b2a3d427182b72f8694c1e589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d158538b2a3d427182b72f8694c1e589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36813 } } peers { permanent_uuid: "1555431512ad4290b1cd00b2a315816e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34461 } }
I20260929 01:02:36.992892 7782 consensus_queue.cc:260] T 0e747d853a07402880d95b212251a93a P 1555431512ad4290b1cd00b2a315816e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d158538b2a3d427182b72f8694c1e589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36813 } } peers { permanent_uuid: "1555431512ad4290b1cd00b2a315816e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34461 } }
I20260929 01:02:36.993325 7781 ts_tablet_manager.cc:1428] T 0e747d853a07402880d95b212251a93a P d158538b2a3d427182b72f8694c1e589: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260929 01:02:36.993680 7611 heartbeater.cc:503] Master 127.0.98.62:44771 was elected leader, sending a full tablet report...
I20260929 01:02:36.996111 7782 ts_tablet_manager.cc:1428] T 0e747d853a07402880d95b212251a93a P 1555431512ad4290b1cd00b2a315816e: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260929 01:02:36.997828 7748 heartbeater.cc:503] Master 127.0.98.62:44771 was elected leader, sending a full tablet report...
W20260929 01:02:37.010340 7476 tablet.cc:2369] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:02:37.023074 7787 raft_consensus.cc:482] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:02:37.023300 7787 raft_consensus.cc:504] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d158538b2a3d427182b72f8694c1e589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36813 } } peers { permanent_uuid: "1555431512ad4290b1cd00b2a315816e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34461 } }
I20260929 01:02:37.024425 7787 leader_election.cc:288] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d158538b2a3d427182b72f8694c1e589 (127.0.98.2:42313), 1555431512ad4290b1cd00b2a315816e (127.0.98.3:34461)
I20260929 01:02:37.028627 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:37.024750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46877 (local address 127.0.98.2:42313)
0929 01:02:37.024883 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:37.024888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:37.024924 (+ 36us) server_negotiation.cc:407] Connection header received
0929 01:02:37.025000 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:37.025005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:37.025057 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:37.025140 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:37.025516 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:37.026325 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:37.027676 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:37.028081 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:37.028203 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:37.028265 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:37.028442 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 01:02:37.028511 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 01:02:37.028630 7478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:37.024626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42313 (local address 127.0.98.1:46877)
0929 01:02:37.024733 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:37.024748 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:37.024836 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:37.025178 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:37.025183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:37.025210 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:37.025389 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:37.025397 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:37.026448 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:37.026451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:37.027414 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:37.027424 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:37.028202 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:37.028205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:37.028308 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:37.028335 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 01:02:37.028428 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 01:02:37.028497 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":44,"spinlock_wait_cycles":8960}
I20260929 01:02:37.028712 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:37.024766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52663 (local address 127.0.98.3:34461)
0929 01:02:37.024898 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:37.024911 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:37.024931 (+ 20us) server_negotiation.cc:407] Connection header received
0929 01:02:37.025052 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:37.025055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:37.025111 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:37.025184 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:37.025516 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:37.026377 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:37.027613 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:37.028080 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:37.028224 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:37.028281 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:37.028515 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 01:02:37.028585 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 01:02:37.028753 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:37.024642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34461 (local address 127.0.98.1:52663)
0929 01:02:37.024874 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:37.024887 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:37.024944 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:37.025204 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:37.025206 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:37.025219 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:37.025407 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:37.025412 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:37.026489 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:37.026493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:37.027440 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:37.027451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:37.028233 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:37.028236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:37.028385 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:37.028404 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 01:02:37.028496 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 01:02:37.028564 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":169,"spinlock_wait_cycles":768}
I20260929 01:02:37.029320 7564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e747d853a07402880d95b212251a93a" candidate_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d158538b2a3d427182b72f8694c1e589" is_pre_election: true
I20260929 01:02:37.029422 7701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e747d853a07402880d95b212251a93a" candidate_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1555431512ad4290b1cd00b2a315816e" is_pre_election: true
I20260929 01:02:37.029642 7701 raft_consensus.cc:2454] T 0e747d853a07402880d95b212251a93a P 1555431512ad4290b1cd00b2a315816e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1e8fd0fcac4b96bf8d5c75404d2e23 in term 0.
I20260929 01:02:37.029642 7564 raft_consensus.cc:2454] T 0e747d853a07402880d95b212251a93a P d158538b2a3d427182b72f8694c1e589 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1e8fd0fcac4b96bf8d5c75404d2e23 in term 0.
I20260929 01:02:37.030068 7363 leader_election.cc:302] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1555431512ad4290b1cd00b2a315816e, fb1e8fd0fcac4b96bf8d5c75404d2e23; no voters:
I20260929 01:02:37.030282 7787 raft_consensus.cc:2781] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:02:37.030385 7787 raft_consensus.cc:482] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:02:37.030441 7787 raft_consensus.cc:3037] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:37.031477 7787 raft_consensus.cc:504] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d158538b2a3d427182b72f8694c1e589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36813 } } peers { permanent_uuid: "1555431512ad4290b1cd00b2a315816e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34461 } }
I20260929 01:02:37.031801 7787 leader_election.cc:288] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23 [CANDIDATE]: Term 1 election: Requested vote from peers d158538b2a3d427182b72f8694c1e589 (127.0.98.2:42313), 1555431512ad4290b1cd00b2a315816e (127.0.98.3:34461)
I20260929 01:02:37.032123 7701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e747d853a07402880d95b212251a93a" candidate_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1555431512ad4290b1cd00b2a315816e"
I20260929 01:02:37.032164 7564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e747d853a07402880d95b212251a93a" candidate_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d158538b2a3d427182b72f8694c1e589"
I20260929 01:02:37.032233 7701 raft_consensus.cc:3037] T 0e747d853a07402880d95b212251a93a P 1555431512ad4290b1cd00b2a315816e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:37.032348 7564 raft_consensus.cc:3037] T 0e747d853a07402880d95b212251a93a P d158538b2a3d427182b72f8694c1e589 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:37.033300 7701 raft_consensus.cc:2454] T 0e747d853a07402880d95b212251a93a P 1555431512ad4290b1cd00b2a315816e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1e8fd0fcac4b96bf8d5c75404d2e23 in term 1.
I20260929 01:02:37.033612 7363 leader_election.cc:302] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1555431512ad4290b1cd00b2a315816e, fb1e8fd0fcac4b96bf8d5c75404d2e23; no voters:
I20260929 01:02:37.033782 7787 raft_consensus.cc:2781] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:37.033998 7787 raft_consensus.cc:686] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 1 LEADER]: Becoming Leader. State: Replica: fb1e8fd0fcac4b96bf8d5c75404d2e23, State: Running, Role: LEADER
I20260929 01:02:37.034018 7564 raft_consensus.cc:2454] T 0e747d853a07402880d95b212251a93a P d158538b2a3d427182b72f8694c1e589 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1e8fd0fcac4b96bf8d5c75404d2e23 in term 1.
I20260929 01:02:37.034325 7787 consensus_queue.cc:237] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d158538b2a3d427182b72f8694c1e589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36813 } } peers { permanent_uuid: "1555431512ad4290b1cd00b2a315816e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34461 } }
I20260929 01:02:37.037648 6510 catalog_manager.cc:5495] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb1e8fd0fcac4b96bf8d5c75404d2e23 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d158538b2a3d427182b72f8694c1e589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1555431512ad4290b1cd00b2a315816e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34461 } health_report { overall_health: UNKNOWN } } }
I20260929 01:02:37.049317 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:37.046296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42944 (local address 127.0.98.1:36813)
0929 01:02:37.046459 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:37.046462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:37.046475 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:02:37.046541 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:37.046545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:37.046591 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:37.046673 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:02:37.046924 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:37.047414 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:37.048174 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:37.048378 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:37.048467 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:37.048671 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:02:37.049046 (+ 375us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:02:37.049130 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:37.049179 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 01:02:37.049222 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 01:02:37.054522 7701 raft_consensus.cc:1261] T 0e747d853a07402880d95b212251a93a P 1555431512ad4290b1cd00b2a315816e [term 1 FOLLOWER]: Refusing update from remote peer fb1e8fd0fcac4b96bf8d5c75404d2e23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:37.054543 7564 raft_consensus.cc:1261] T 0e747d853a07402880d95b212251a93a P d158538b2a3d427182b72f8694c1e589 [term 1 FOLLOWER]: Refusing update from remote peer fb1e8fd0fcac4b96bf8d5c75404d2e23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:37.055256 7787 consensus_queue.cc:1035] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d158538b2a3d427182b72f8694c1e589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:37.055447 7790 consensus_queue.cc:1035] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1555431512ad4290b1cd00b2a315816e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:37.068513 7794 mvcc.cc:205] Tried to move back new op lower bound from 7334476828889034752 to 7334476828815486976. Current Snapshot: MvccSnapshot[applied={T|T < 7334476828889034752}]
I20260929 01:02:37.070856 7795 mvcc.cc:205] Tried to move back new op lower bound from 7334476828889034752 to 7334476828815486976. Current Snapshot: MvccSnapshot[applied={T|T < 7334476828889034752}]
I20260929 01:02:37.071403 6510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:37.071558 6510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:37.075300 6510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d621ac6ce2e468f8451220161009672: Sending DeleteTablet for 3 replicas of tablet 0e747d853a07402880d95b212251a93a
I20260929 01:02:37.076138 7407 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e747d853a07402880d95b212251a93a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42936
I20260929 01:02:37.076246 7681 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e747d853a07402880d95b212251a93a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40964
I20260929 01:02:37.077189 7544 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e747d853a07402880d95b212251a93a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60872
I20260929 01:02:37.077658 7804 tablet_replica.cc:330] T 0e747d853a07402880d95b212251a93a P 1555431512ad4290b1cd00b2a315816e: stopping tablet replica
I20260929 01:02:37.078331 7804 raft_consensus.cc:2229] T 0e747d853a07402880d95b212251a93a P 1555431512ad4290b1cd00b2a315816e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:37.079227 7804 raft_consensus.cc:2258] T 0e747d853a07402880d95b212251a93a P 1555431512ad4290b1cd00b2a315816e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:37.079665 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 6479
I20260929 01:02:37.082088 7804 ts_tablet_manager.cc:1905] T 0e747d853a07402880d95b212251a93a P 1555431512ad4290b1cd00b2a315816e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:37.085582 7804 ts_tablet_manager.cc:1918] T 0e747d853a07402880d95b212251a93a P 1555431512ad4290b1cd00b2a315816e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:02:37.085741 7804 log.cc:1196] T 0e747d853a07402880d95b212251a93a P 1555431512ad4290b1cd00b2a315816e: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/0e747d853a07402880d95b212251a93a
I20260929 01:02:37.086105 7804 ts_tablet_manager.cc:1939] T 0e747d853a07402880d95b212251a93a P 1555431512ad4290b1cd00b2a315816e: Deleting consensus metadata
W20260929 01:02:37.087524 7637 connection.cc:535] server connection from 127.0.0.1:40964 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:02:37.087801 7803 tablet_replica.cc:330] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23: stopping tablet replica
I20260929 01:02:37.088179 7803 raft_consensus.cc:2229] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:02:37.088526 7803 raft_consensus.cc:2258] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:37.092074 7803 ts_tablet_manager.cc:1905] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:37.093067 7805 tablet_replica.cc:330] T 0e747d853a07402880d95b212251a93a P d158538b2a3d427182b72f8694c1e589: stopping tablet replica
I20260929 01:02:37.093403 7805 raft_consensus.cc:2229] T 0e747d853a07402880d95b212251a93a P d158538b2a3d427182b72f8694c1e589 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:37.093758 7805 raft_consensus.cc:2258] T 0e747d853a07402880d95b212251a93a P d158538b2a3d427182b72f8694c1e589 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:37.094672 7803 ts_tablet_manager.cc:1918] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:02:37.094798 7803 log.cc:1196] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-0/wal/wals/0e747d853a07402880d95b212251a93a
I20260929 01:02:37.095203 7803 ts_tablet_manager.cc:1939] T 0e747d853a07402880d95b212251a93a P fb1e8fd0fcac4b96bf8d5c75404d2e23: Deleting consensus metadata
W20260929 01:02:37.096202 7363 connection.cc:441] server connection from 127.0.0.1:42936 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42936 (request call id 1) could send its response
I20260929 01:02:37.097311 7805 ts_tablet_manager.cc:1905] T 0e747d853a07402880d95b212251a93a P d158538b2a3d427182b72f8694c1e589: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:37.101686 7805 ts_tablet_manager.cc:1918] T 0e747d853a07402880d95b212251a93a P d158538b2a3d427182b72f8694c1e589: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:02:37.101785 7805 log.cc:1196] T 0e747d853a07402880d95b212251a93a P d158538b2a3d427182b72f8694c1e589: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/wal/wals/0e747d853a07402880d95b212251a93a
I20260929 01:02:37.102118 7805 ts_tablet_manager.cc:1939] T 0e747d853a07402880d95b212251a93a P d158538b2a3d427182b72f8694c1e589: Deleting consensus metadata
W20260929 01:02:37.103365 7497 connection.cc:441] server connection from 127.0.0.1:60872 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60872 (request call id 1) could send its response
I20260929 01:02:37.103549 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:37.079197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58390 (local address 127.0.98.61:40081)
0929 01:02:37.079316 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:37.079321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:37.089017 (+ 9696us) server_negotiation.cc:407] Connection header received
0929 01:02:37.089209 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:37.089212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:37.089262 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:37.089317 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:37.090777 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:37.091629 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:37.094079 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:37.094459 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:37.094533 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:37.098921 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:37.098925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:37.098927 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:37.098952 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:37.100950 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:37.101774 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:37.101778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:37.101780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:37.101872 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:37.102241 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:37.102244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:37.102246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:37.102451 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:37.102549 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:37.102952 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 01:02:37.103389 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 01:02:37.105710 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:37.091469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58394 (local address 127.0.98.61:40081)
0929 01:02:37.091569 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:37.091573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:37.093005 (+ 1432us) server_negotiation.cc:407] Connection header received
0929 01:02:37.093238 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:37.093241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:37.093287 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:37.093335 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:37.094466 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:37.095252 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:37.096471 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:37.096770 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:37.096839 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:37.098425 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:37.098429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:37.098432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:37.098456 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:37.101306 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:37.104261 (+ 2955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:37.104267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:37.104269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:37.104354 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:37.104658 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:37.104662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:37.104663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:37.104885 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:37.105005 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:37.105393 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 01:02:37.105520 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 01:02:37.108084 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:37.092135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60124 (local address 127.0.98.60:36721)
0929 01:02:37.092271 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:37.092275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:37.095270 (+ 2995us) server_negotiation.cc:407] Connection header received
0929 01:02:37.099054 (+ 3784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:37.099058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:37.099118 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:37.099180 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:37.100178 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:37.100903 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:37.102180 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:37.102507 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:37.102582 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:37.104477 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:37.104481 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:37.104484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:37.104514 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:37.106231 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:37.106736 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:37.106739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:37.106741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:37.106928 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:37.107225 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:37.107231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:37.107234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:37.107435 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:37.107533 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:37.107823 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 01:02:37.107933 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 01:02:37.113062 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:37.078944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60114 (local address 127.0.98.60:36721)
0929 01:02:37.079054 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:37.079059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:37.105506 (+ 26447us) server_negotiation.cc:407] Connection header received
0929 01:02:37.105658 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:37.105661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:37.105708 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:37.105756 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:37.106485 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:37.107280 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:37.108405 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:37.108698 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:37.108812 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:37.110043 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:37.110046 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:37.110048 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:37.110071 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:37.111483 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:37.111877 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:37.111880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:37.111882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:37.111960 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:37.112226 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:37.112234 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:37.112236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:37.112407 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:37.112546 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:37.112766 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 01:02:37.112887 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 01:02:38.067111 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.066670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.98.3:35213)
0929 01:02:38.066951 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.067015 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":65,"threads_started":1}
I20260929 01:02:38.067111 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.066604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.98.1:57851)
0929 01:02:38.066899 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.067001 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":73,"threads_started":1}
W20260929 01:02:38.067376 7748 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44771 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
W20260929 01:02:38.067641 7474 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44771 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
I20260929 01:02:38.068296 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.067864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.98.2:49915)
0929 01:02:38.068080 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.068153 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":50,"threads_started":1}
W20260929 01:02:38.068554 7611 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44771 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
I20260929 01:02:38.575555 7817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1d621ac6ce2e468f8451220161009672)
I20260929 01:02:38.575712 7817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: false } }
I20260929 01:02:38.576177 7817 leader_election.cc:288] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d621ac6ce2e468f8451220161009672 (127.0.98.62:44771), 7bcff1ee79b34e0cb34c599ba53620e9 (127.0.98.61:40081)
I20260929 01:02:38.576429 7818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1d621ac6ce2e468f8451220161009672)
I20260929 01:02:38.576597 7818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: false } }
I20260929 01:02:38.576910 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.576393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.0.1:34016)
0929 01:02:38.576707 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.576806 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":74,"threads_started":1}
W20260929 01:02:38.577109 7280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
I20260929 01:02:38.577795 7818 leader_election.cc:288] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d621ac6ce2e468f8451220161009672 (127.0.98.62:44771), 1a6f4f441458470689115fea13835036 (127.0.98.60:36721)
I20260929 01:02:38.578092 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.577836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.0.1:34018)
0929 01:02:38.577968 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.578023 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 01:02:38.578292 7280 leader_election.cc:334] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1d621ac6ce2e468f8451220161009672 (127.0.98.62:44771): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
I20260929 01:02:38.579021 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.578399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.0.1:34026)
0929 01:02:38.578873 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.578939 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":54,"threads_started":1}
W20260929 01:02:38.579250 7097 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
I20260929 01:02:38.580143 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.579949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.0.1:34042)
0929 01:02:38.580033 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.580082 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260929 01:02:38.580309 7097 leader_election.cc:334] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1d621ac6ce2e468f8451220161009672 (127.0.98.62:44771): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
I20260929 01:02:38.581393 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.576434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40081 (local address 127.0.0.1:58398)
0929 01:02:38.576850 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.576875 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:38.577081 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:38.577391 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:38.577396 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:38.577417 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:38.577522 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:38.577528 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:38.578980 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:38.578983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:38.580238 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:38.580250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:38.580972 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:38.580976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:38.581062 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:38.581084 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 01:02:38.581198 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 01:02:38.581272 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":18,"thread_start_us":58,"threads_started":1}
I20260929 01:02:38.581391 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.576462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58398 (local address 127.0.98.61:40081)
0929 01:02:38.576771 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:38.576777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:38.576973 (+ 196us) server_negotiation.cc:407] Connection header received
0929 01:02:38.577212 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:38.577216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:38.577268 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:38.577368 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:38.577643 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:38.578863 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:38.580426 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:38.580849 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:38.580968 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:38.581027 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:38.581216 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 01:02:38.581280 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":64,"threads_started":1}
I20260929 01:02:38.581887 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.578102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36721 (local address 127.0.0.1:60130)
0929 01:02:38.578575 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.578586 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:38.578664 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:38.578931 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:38.578934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:38.578953 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:38.579039 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:38.579045 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:38.579932 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:38.579936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:38.580897 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:38.580905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:38.581533 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:38.581536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:38.581625 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:38.581647 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 01:02:38.581710 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 01:02:38.581773 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":78,"threads_started":1}
I20260929 01:02:38.581853 7120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a6f4f441458470689115fea13835036" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" is_pre_election: true
I20260929 01:02:38.582002 7120 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a6f4f441458470689115fea13835036 in term 1.
I20260929 01:02:38.582259 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.578224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60130 (local address 127.0.98.60:36721)
0929 01:02:38.578459 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:38.578468 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:38.578653 (+ 185us) server_negotiation.cc:407] Connection header received
0929 01:02:38.578760 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:38.578763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:38.578836 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:38.578922 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:38.579183 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:38.579788 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:38.581017 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:38.581419 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:38.581533 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:38.581584 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:38.582075 (+ 491us) server_negotiation.cc:299] Negotiation successful
0929 01:02:38.582146 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":179,"thread_start_us":126,"threads_started":1}
I20260929 01:02:38.582275 7279 leader_election.cc:302] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a6f4f441458470689115fea13835036, 7bcff1ee79b34e0cb34c599ba53620e9; no voters: 1d621ac6ce2e468f8451220161009672
I20260929 01:02:38.582463 7817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 01:02:38.582535 7817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1d621ac6ce2e468f8451220161009672)
I20260929 01:02:38.582608 7817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:02:38.582635 7304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1a6f4f441458470689115fea13835036" is_pre_election: true
I20260929 01:02:38.583659 7817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: false } }
I20260929 01:02:38.583922 7304 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7bcff1ee79b34e0cb34c599ba53620e9 in current term 2: Already voted for candidate 1a6f4f441458470689115fea13835036 in this term.
I20260929 01:02:38.584039 7817 leader_election.cc:288] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [CANDIDATE]: Term 2 election: Requested vote from peers 1d621ac6ce2e468f8451220161009672 (127.0.98.62:44771), 7bcff1ee79b34e0cb34c599ba53620e9 (127.0.98.61:40081)
I20260929 01:02:38.584384 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.584139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.0.1:34052)
0929 01:02:38.584258 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.584309 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 01:02:38.584443 7120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a6f4f441458470689115fea13835036" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7bcff1ee79b34e0cb34c599ba53620e9"
I20260929 01:02:38.584559 7120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:02:38.584805 7095 leader_election.cc:302] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7bcff1ee79b34e0cb34c599ba53620e9; no voters: 1a6f4f441458470689115fea13835036, 1d621ac6ce2e468f8451220161009672
I20260929 01:02:38.585602 7120 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a6f4f441458470689115fea13835036 in term 2.
I20260929 01:02:38.585712 7818 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:02:38.585866 7279 leader_election.cc:302] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a6f4f441458470689115fea13835036, 7bcff1ee79b34e0cb34c599ba53620e9; no voters:
I20260929 01:02:38.586025 7817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:02:38.586223 7817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [term 2 LEADER]: Becoming Leader. State: Replica: 1a6f4f441458470689115fea13835036, State: Running, Role: LEADER
I20260929 01:02:38.586370 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.586125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.0.1:34054)
0929 01:02:38.586265 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.586309 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260929 01:02:38.586488 7280 leader_election.cc:334] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1d621ac6ce2e468f8451220161009672 (127.0.98.62:44771): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
I20260929 01:02:38.586508 7817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: false } }
I20260929 01:02:38.587699 7827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a6f4f441458470689115fea13835036. Latest consensus state: current_term: 2 leader_uuid: "1a6f4f441458470689115fea13835036" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: false } } }
I20260929 01:02:38.587785 7827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:38.588053 7829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:02:38.590211 7829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:02:38.590605 7829 catalog_manager.cc:1259] Loaded cluster ID: 3792c0e2458b4d2088777c90f9a31006
I20260929 01:02:38.590673 7829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:02:38.591245 7829 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:02:38.591629 7829 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036: Loaded TSK: 0
I20260929 01:02:38.592331 7829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:02:38.619431 7294 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:02:38.623487 7294 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:02:38.626543 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.626231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.0.1:34070)
0929 01:02:38.626386 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.626466 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 01:02:38.626866 7120 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [term 2 FOLLOWER]: Refusing update from remote peer 1a6f4f441458470689115fea13835036: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 01:02:38.627195 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.626976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.0.1:34082)
0929 01:02:38.627106 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.627144 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260929 01:02:38.627463 7280 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 -> Peer 1d621ac6ce2e468f8451220161009672 (127.0.98.62:44771): Couldn't send request to peer 1d621ac6ce2e468f8451220161009672. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 01:02:38.627624 7817 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 01:02:38.629760 7818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a6f4f441458470689115fea13835036. Latest consensus state: current_term: 2 leader_uuid: "1a6f4f441458470689115fea13835036" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: false } } }
I20260929 01:02:38.629925 7818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:02:38.632025 7827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "1a6f4f441458470689115fea13835036" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: false } } }
I20260929 01:02:38.632148 7827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036 [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:38.632383 7831 mvcc.cc:205] Tried to move back new op lower bound from 7334476835329585152 to 7334476835172986880. Current Snapshot: MvccSnapshot[applied={T|T < 7334476835329585152}]
I20260929 01:02:38.632552 7818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "1a6f4f441458470689115fea13835036" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1d621ac6ce2e468f8451220161009672" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } peers { permanent_uuid: "7bcff1ee79b34e0cb34c599ba53620e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40081 } attrs { promote: false } } peers { permanent_uuid: "1a6f4f441458470689115fea13835036" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36721 } attrs { promote: false } } }
I20260929 01:02:38.632684 7818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bcff1ee79b34e0cb34c599ba53620e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:02:38.643505 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.639386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42313 (local address 127.0.0.1:60878)
0929 01:02:38.639541 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.639558 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:38.639642 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:38.639974 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:38.639980 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:38.640001 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:38.640103 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:38.640109 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:38.641107 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:38.641111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:38.642272 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:38.642284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:38.643098 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:38.643101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:38.643192 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:38.643221 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 01:02:38.643268 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 01:02:38.643341 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":1}
I20260929 01:02:38.644142 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.639477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60878 (local address 127.0.98.2:42313)
0929 01:02:38.639736 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:38.639741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:38.639757 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:02:38.639813 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:38.639819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:38.639871 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:38.639958 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:38.640207 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:38.640963 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:38.642377 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:38.642960 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:38.643795 (+ 835us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:38.643862 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:38.643902 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 01:02:38.643992 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":101,"threads_started":1}
I20260929 01:02:38.644717 7544 tablet_service.cc:1467] Processing CreateTablet for tablet 0bbd4824bea042fd9d14b6c157f6cde1 (DEFAULT_TABLE table=test-table [id=42e0fecdfd444e76b0360e2acd319082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:38.645049 7544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bbd4824bea042fd9d14b6c157f6cde1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:38.645220 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.639648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34461 (local address 127.0.0.1:40974)
0929 01:02:38.639784 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.639797 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:38.639869 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:38.640372 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:38.640377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:38.640397 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:38.640479 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:38.640487 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:38.642357 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:38.642359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:38.643573 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:38.643584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:38.644805 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:38.644811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:38.644927 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:38.644951 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 01:02:38.645005 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 01:02:38.645074 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 01:02:38.645919 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.639877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40974 (local address 127.0.98.3:34461)
0929 01:02:38.640130 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:38.640137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:38.640152 (+ 15us) server_negotiation.cc:407] Connection header received
0929 01:02:38.640204 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:38.640207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:38.640254 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:38.640597 (+ 343us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:38.640713 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:38.642230 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:38.643733 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:38.644676 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:38.645551 (+ 875us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:38.645613 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:38.645673 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 01:02:38.645760 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":71,"threads_started":1}
I20260929 01:02:38.648223 7840 tablet_bootstrap.cc:492] T 0bbd4824bea042fd9d14b6c157f6cde1 P d158538b2a3d427182b72f8694c1e589: Bootstrap starting.
I20260929 01:02:38.649403 7840 tablet_bootstrap.cc:654] T 0bbd4824bea042fd9d14b6c157f6cde1 P d158538b2a3d427182b72f8694c1e589: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:38.650653 7840 tablet_bootstrap.cc:492] T 0bbd4824bea042fd9d14b6c157f6cde1 P d158538b2a3d427182b72f8694c1e589: No bootstrap required, opened a new log
I20260929 01:02:38.650692 7681 tablet_service.cc:1467] Processing CreateTablet for tablet 0bbd4824bea042fd9d14b6c157f6cde1 (DEFAULT_TABLE table=test-table [id=42e0fecdfd444e76b0360e2acd319082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:38.650746 7840 ts_tablet_manager.cc:1397] T 0bbd4824bea042fd9d14b6c157f6cde1 P d158538b2a3d427182b72f8694c1e589: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 01:02:38.650975 7681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bbd4824bea042fd9d14b6c157f6cde1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:38.651243 7840 raft_consensus.cc:348] T 0bbd4824bea042fd9d14b6c157f6cde1 P d158538b2a3d427182b72f8694c1e589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d158538b2a3d427182b72f8694c1e589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "1555431512ad4290b1cd00b2a315816e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34461 } } peers { permanent_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36813 } }
I20260929 01:02:38.651376 7840 raft_consensus.cc:374] T 0bbd4824bea042fd9d14b6c157f6cde1 P d158538b2a3d427182b72f8694c1e589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:38.651407 7840 raft_consensus.cc:729] T 0bbd4824bea042fd9d14b6c157f6cde1 P d158538b2a3d427182b72f8694c1e589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d158538b2a3d427182b72f8694c1e589, State: Initialized, Role: FOLLOWER
I20260929 01:02:38.651546 7840 consensus_queue.cc:260] T 0bbd4824bea042fd9d14b6c157f6cde1 P d158538b2a3d427182b72f8694c1e589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d158538b2a3d427182b72f8694c1e589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "1555431512ad4290b1cd00b2a315816e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34461 } } peers { permanent_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36813 } }
I20260929 01:02:38.651811 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.640443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36813 (local address 127.0.0.1:42958)
0929 01:02:38.640673 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.640916 (+ 243us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:38.641016 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:38.641430 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:38.641433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:38.641451 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:38.641549 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:38.641557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:38.642484 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:38.642487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:38.643826 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:38.643833 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:38.651418 (+ 7585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:38.651422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:38.651512 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:38.651535 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 01:02:38.651598 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 01:02:38.651674 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":90,"threads_started":1}
I20260929 01:02:38.651906 7840 ts_tablet_manager.cc:1428] T 0bbd4824bea042fd9d14b6c157f6cde1 P d158538b2a3d427182b72f8694c1e589: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:38.652637 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.652269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.98.2:47043)
0929 01:02:38.652498 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.652553 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20260929 01:02:38.653225 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.653032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.98.2:38993)
0929 01:02:38.653126 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.653169 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 01:02:38.653553 7844 tablet_bootstrap.cc:492] T 0bbd4824bea042fd9d14b6c157f6cde1 P 1555431512ad4290b1cd00b2a315816e: Bootstrap starting.
I20260929 01:02:38.654500 7844 tablet_bootstrap.cc:654] T 0bbd4824bea042fd9d14b6c157f6cde1 P 1555431512ad4290b1cd00b2a315816e: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:38.654901 7610 heartbeater.cc:503] Master 127.0.98.60:36721 was elected leader, sending a full tablet report...
I20260929 01:02:38.655161 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.640774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42958 (local address 127.0.98.1:36813)
0929 01:02:38.641008 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:38.641012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:38.641027 (+ 15us) server_negotiation.cc:407] Connection header received
0929 01:02:38.641149 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:38.641153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:38.641216 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:38.641302 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:38.641709 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:38.642395 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:38.650396 (+ 8001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:38.651020 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:38.651128 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:38.652046 (+ 918us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:38.652088 (+ 42us) server_negotiation.cc:299] Negotiation successful
0929 01:02:38.655016 (+ 2928us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":79,"threads_started":1}
I20260929 01:02:38.655673 7844 tablet_bootstrap.cc:492] T 0bbd4824bea042fd9d14b6c157f6cde1 P 1555431512ad4290b1cd00b2a315816e: No bootstrap required, opened a new log
I20260929 01:02:38.655615 7407 tablet_service.cc:1467] Processing CreateTablet for tablet 0bbd4824bea042fd9d14b6c157f6cde1 (DEFAULT_TABLE table=test-table [id=42e0fecdfd444e76b0360e2acd319082]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:38.655753 7844 ts_tablet_manager.cc:1397] T 0bbd4824bea042fd9d14b6c157f6cde1 P 1555431512ad4290b1cd00b2a315816e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:38.655889 7407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bbd4824bea042fd9d14b6c157f6cde1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:38.656185 7844 raft_consensus.cc:348] T 0bbd4824bea042fd9d14b6c157f6cde1 P 1555431512ad4290b1cd00b2a315816e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d158538b2a3d427182b72f8694c1e589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "1555431512ad4290b1cd00b2a315816e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34461 } } peers { permanent_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36813 } }
I20260929 01:02:38.656301 7844 raft_consensus.cc:374] T 0bbd4824bea042fd9d14b6c157f6cde1 P 1555431512ad4290b1cd00b2a315816e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:38.656340 7844 raft_consensus.cc:729] T 0bbd4824bea042fd9d14b6c157f6cde1 P 1555431512ad4290b1cd00b2a315816e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1555431512ad4290b1cd00b2a315816e, State: Initialized, Role: FOLLOWER
I20260929 01:02:38.656466 7844 consensus_queue.cc:260] T 0bbd4824bea042fd9d14b6c157f6cde1 P 1555431512ad4290b1cd00b2a315816e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d158538b2a3d427182b72f8694c1e589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "1555431512ad4290b1cd00b2a315816e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34461 } } peers { permanent_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36813 } }
I20260929 01:02:38.656782 7844 ts_tablet_manager.cc:1428] T 0bbd4824bea042fd9d14b6c157f6cde1 P 1555431512ad4290b1cd00b2a315816e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:38.658051 7747 heartbeater.cc:503] Master 127.0.98.60:36721 was elected leader, sending a full tablet report...
I20260929 01:02:38.658252 7847 tablet_bootstrap.cc:492] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23: Bootstrap starting.
I20260929 01:02:38.659214 7847 tablet_bootstrap.cc:654] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:38.660284 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.658572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.98.3:37401)
0929 01:02:38.660145 (+ 1573us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.660199 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1511,"thread_start_us":86,"threads_started":1}
W20260929 01:02:38.660787 7750 tablet.cc:2369] T 0bbd4824bea042fd9d14b6c157f6cde1 P 1555431512ad4290b1cd00b2a315816e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:02:38.661417 7847 tablet_bootstrap.cc:492] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23: No bootstrap required, opened a new log
I20260929 01:02:38.661499 7847 ts_tablet_manager.cc:1397] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 01:02:38.661839 7847 raft_consensus.cc:348] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d158538b2a3d427182b72f8694c1e589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "1555431512ad4290b1cd00b2a315816e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34461 } } peers { permanent_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36813 } }
I20260929 01:02:38.661950 7847 raft_consensus.cc:374] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:38.661988 7847 raft_consensus.cc:729] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1e8fd0fcac4b96bf8d5c75404d2e23, State: Initialized, Role: FOLLOWER
I20260929 01:02:38.662108 7847 consensus_queue.cc:260] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d158538b2a3d427182b72f8694c1e589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "1555431512ad4290b1cd00b2a315816e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34461 } } peers { permanent_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36813 } }
I20260929 01:02:38.662393 7847 ts_tablet_manager.cc:1428] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:38.663182 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.662765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.98.1:38775)
0929 01:02:38.663033 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.663089 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":124,"threads_started":1}
I20260929 01:02:38.663945 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.663742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.98.1:55279)
0929 01:02:38.663833 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.663886 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 01:02:38.664019 7473 heartbeater.cc:503] Master 127.0.98.60:36721 was elected leader, sending a full tablet report...
W20260929 01:02:38.709918 7613 tablet.cc:2369] T 0bbd4824bea042fd9d14b6c157f6cde1 P d158538b2a3d427182b72f8694c1e589: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:02:38.812947 7850 raft_consensus.cc:482] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:02:38.813098 7850 raft_consensus.cc:504] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d158538b2a3d427182b72f8694c1e589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "1555431512ad4290b1cd00b2a315816e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34461 } } peers { permanent_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36813 } }
I20260929 01:02:38.813596 7850 leader_election.cc:288] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d158538b2a3d427182b72f8694c1e589 (127.0.98.2:42313), 1555431512ad4290b1cd00b2a315816e (127.0.98.3:34461)
I20260929 01:02:38.814103 7701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bbd4824bea042fd9d14b6c157f6cde1" candidate_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1555431512ad4290b1cd00b2a315816e" is_pre_election: true
I20260929 01:02:38.814105 7564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bbd4824bea042fd9d14b6c157f6cde1" candidate_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d158538b2a3d427182b72f8694c1e589" is_pre_election: true
I20260929 01:02:38.814268 7701 raft_consensus.cc:2454] T 0bbd4824bea042fd9d14b6c157f6cde1 P 1555431512ad4290b1cd00b2a315816e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1e8fd0fcac4b96bf8d5c75404d2e23 in term 0.
I20260929 01:02:38.814278 7564 raft_consensus.cc:2454] T 0bbd4824bea042fd9d14b6c157f6cde1 P d158538b2a3d427182b72f8694c1e589 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1e8fd0fcac4b96bf8d5c75404d2e23 in term 0.
I20260929 01:02:38.814653 7363 leader_election.cc:302] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1555431512ad4290b1cd00b2a315816e, fb1e8fd0fcac4b96bf8d5c75404d2e23; no voters:
I20260929 01:02:38.814863 7850 raft_consensus.cc:2781] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:02:38.814924 7850 raft_consensus.cc:482] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:02:38.814991 7850 raft_consensus.cc:3037] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:38.816061 7850 raft_consensus.cc:504] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d158538b2a3d427182b72f8694c1e589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "1555431512ad4290b1cd00b2a315816e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34461 } } peers { permanent_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36813 } }
I20260929 01:02:38.816411 7850 leader_election.cc:288] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23 [CANDIDATE]: Term 1 election: Requested vote from peers d158538b2a3d427182b72f8694c1e589 (127.0.98.2:42313), 1555431512ad4290b1cd00b2a315816e (127.0.98.3:34461)
I20260929 01:02:38.816715 7564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bbd4824bea042fd9d14b6c157f6cde1" candidate_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d158538b2a3d427182b72f8694c1e589"
I20260929 01:02:38.816735 7701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bbd4824bea042fd9d14b6c157f6cde1" candidate_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1555431512ad4290b1cd00b2a315816e"
I20260929 01:02:38.816820 7701 raft_consensus.cc:3037] T 0bbd4824bea042fd9d14b6c157f6cde1 P 1555431512ad4290b1cd00b2a315816e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:38.816845 7564 raft_consensus.cc:3037] T 0bbd4824bea042fd9d14b6c157f6cde1 P d158538b2a3d427182b72f8694c1e589 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:38.817670 7701 raft_consensus.cc:2454] T 0bbd4824bea042fd9d14b6c157f6cde1 P 1555431512ad4290b1cd00b2a315816e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1e8fd0fcac4b96bf8d5c75404d2e23 in term 1.
I20260929 01:02:38.817854 7564 raft_consensus.cc:2454] T 0bbd4824bea042fd9d14b6c157f6cde1 P d158538b2a3d427182b72f8694c1e589 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1e8fd0fcac4b96bf8d5c75404d2e23 in term 1.
I20260929 01:02:38.817935 7363 leader_election.cc:302] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1555431512ad4290b1cd00b2a315816e, fb1e8fd0fcac4b96bf8d5c75404d2e23; no voters:
I20260929 01:02:38.818192 7850 raft_consensus.cc:2781] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:38.818372 7850 raft_consensus.cc:686] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 1 LEADER]: Becoming Leader. State: Replica: fb1e8fd0fcac4b96bf8d5c75404d2e23, State: Running, Role: LEADER
I20260929 01:02:38.818526 7850 consensus_queue.cc:237] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d158538b2a3d427182b72f8694c1e589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } } peers { permanent_uuid: "1555431512ad4290b1cd00b2a315816e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34461 } } peers { permanent_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36813 } }
I20260929 01:02:38.820186 7293 catalog_manager.cc:5495] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb1e8fd0fcac4b96bf8d5c75404d2e23 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d158538b2a3d427182b72f8694c1e589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1555431512ad4290b1cd00b2a315816e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb1e8fd0fcac4b96bf8d5c75404d2e23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36813 } health_report { overall_health: HEALTHY } } }
I20260929 01:02:38.820390 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.819090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.98.1:53893)
0929 01:02:38.820222 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.820292 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 01:02:38.832340 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.829661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42966 (local address 127.0.98.1:36813)
0929 01:02:38.829771 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:38.829777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:38.829794 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:02:38.829831 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:38.829833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:38.829871 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:38.829933 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:02:38.830130 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:38.830674 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:38.831497 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:38.831701 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:38.831782 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:38.831963 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:02:38.832034 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:02:38.832083 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:38.832153 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 01:02:38.832191 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 01:02:38.834247 7701 raft_consensus.cc:1261] T 0bbd4824bea042fd9d14b6c157f6cde1 P 1555431512ad4290b1cd00b2a315816e [term 1 FOLLOWER]: Refusing update from remote peer fb1e8fd0fcac4b96bf8d5c75404d2e23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:38.834254 7564 raft_consensus.cc:1261] T 0bbd4824bea042fd9d14b6c157f6cde1 P d158538b2a3d427182b72f8694c1e589 [term 1 FOLLOWER]: Refusing update from remote peer fb1e8fd0fcac4b96bf8d5c75404d2e23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:38.834720 7850 consensus_queue.cc:1035] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1555431512ad4290b1cd00b2a315816e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:38.834895 7852 consensus_queue.cc:1035] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d158538b2a3d427182b72f8694c1e589" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:38.836596 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.836374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.98.2:51893)
0929 01:02:38.836469 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.836525 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 01:02:38.837371 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.836865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.98.3:40061)
0929 01:02:38.837223 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.837295 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282}
I20260929 01:02:38.840196 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.839884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.98.1:57191)
0929 01:02:38.840077 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.840129 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260929 01:02:38.840407 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.839965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.98.2:43367)
0929 01:02:38.840245 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.840327 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220}
I20260929 01:02:38.840641 7855 mvcc.cc:205] Tried to move back new op lower bound from 7334476836181200896 to 7334476836122169344. Current Snapshot: MvccSnapshot[applied={T|T < 7334476836181200896 or (T in {7334476836181200896})}]
I20260929 01:02:38.842490 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.842184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.98.1:34777)
0929 01:02:38.842373 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.842422 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260929 01:02:38.843925 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.843599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.98.3:34079)
0929 01:02:38.843800 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.843859 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260929 01:02:38.844659 7292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:38.844810 7292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:38.848390 7292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a6f4f441458470689115fea13835036: Sending DeleteTablet for 3 replicas of tablet 0bbd4824bea042fd9d14b6c157f6cde1
I20260929 01:02:38.849170 7544 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bbd4824bea042fd9d14b6c157f6cde1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:38 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:60878
I20260929 01:02:38.849261 7407 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bbd4824bea042fd9d14b6c157f6cde1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:38 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:42958
I20260929 01:02:38.849547 7865 tablet_replica.cc:330] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23: stopping tablet replica
I20260929 01:02:38.849685 7865 raft_consensus.cc:2229] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:02:38.849781 7681 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bbd4824bea042fd9d14b6c157f6cde1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:38 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40974
I20260929 01:02:38.849826 7864 tablet_replica.cc:330] T 0bbd4824bea042fd9d14b6c157f6cde1 P d158538b2a3d427182b72f8694c1e589: stopping tablet replica
I20260929 01:02:38.849887 7865 raft_consensus.cc:2258] T 0bbd4824bea042fd9d14b6c157f6cde1 P fb1e8fd0fcac4b96bf8d5c75404d2e23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:38.849972 7864 raft_consensus.cc:2229] T 0bbd4824bea042fd9d14b6c157f6cde1 P d158538b2a3d427182b72f8694c1e589 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:38.850203 7864 raft_consensus.cc:2258] T 0bbd4824bea042fd9d14b6c157f6cde1 P d158538b2a3d427182b72f8694c1e589 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:38.850400 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 7255
I20260929 01:02:38.853109 7864 ts_tablet_manager.cc:1905] T 0bbd4824bea042fd9d14b6c157f6cde1 P d158538b2a3d427182b72f8694c1e589: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:38.854981 7866 tablet_replica.cc:330] T 0bbd4824bea042fd9d14b6c157f6cde1 P 1555431512ad4290b1cd00b2a315816e: stopping tablet replica
I20260929 01:02:38.855109 7866 raft_consensus.cc:2229] T 0bbd4824bea042fd9d14b6c157f6cde1 P 1555431512ad4290b1cd00b2a315816e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:38.855234 7866 raft_consensus.cc:2258] T 0bbd4824bea042fd9d14b6c157f6cde1 P 1555431512ad4290b1cd00b2a315816e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:38.855876 7866 ts_tablet_manager.cc:1905] T 0bbd4824bea042fd9d14b6c157f6cde1 P 1555431512ad4290b1cd00b2a315816e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:38.856674 7864 ts_tablet_manager.cc:1918] T 0bbd4824bea042fd9d14b6c157f6cde1 P d158538b2a3d427182b72f8694c1e589: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:02:38.856760 7864 log.cc:1196] T 0bbd4824bea042fd9d14b6c157f6cde1 P d158538b2a3d427182b72f8694c1e589: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-1/wal/wals/0bbd4824bea042fd9d14b6c157f6cde1
I20260929 01:02:38.857043 7864 ts_tablet_manager.cc:1939] T 0bbd4824bea042fd9d14b6c157f6cde1 P d158538b2a3d427182b72f8694c1e589: Deleting consensus metadata
I20260929 01:02:38.857812 7278 catalog_manager.cc:4841] TS d158538b2a3d427182b72f8694c1e589 (127.0.98.2:42313): tablet 0bbd4824bea042fd9d14b6c157f6cde1 (table test-table [id=42e0fecdfd444e76b0360e2acd319082]) successfully deleted
I20260929 01:02:38.857898 7866 ts_tablet_manager.cc:1918] T 0bbd4824bea042fd9d14b6c157f6cde1 P 1555431512ad4290b1cd00b2a315816e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:02:38.858003 7866 log.cc:1196] T 0bbd4824bea042fd9d14b6c157f6cde1 P 1555431512ad4290b1cd00b2a315816e: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/0bbd4824bea042fd9d14b6c157f6cde1
I20260929 01:02:38.858278 7866 ts_tablet_manager.cc:1939] T 0bbd4824bea042fd9d14b6c157f6cde1 P 1555431512ad4290b1cd00b2a315816e: Deleting consensus metadata
I20260929 01:02:38.858937 7281 catalog_manager.cc:4841] TS 1555431512ad4290b1cd00b2a315816e (127.0.98.3:34461): tablet 0bbd4824bea042fd9d14b6c157f6cde1 (table test-table [id=42e0fecdfd444e76b0360e2acd319082]) successfully deleted
I20260929 01:02:38.859959 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 7481
I20260929 01:02:38.860356 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:38.860020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36813 (local address 127.0.0.1:42978)
0929 01:02:38.860205 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:38.860263 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260929 01:02:38.861443 7280 catalog_manager.cc:4584] TS fb1e8fd0fcac4b96bf8d5c75404d2e23 (127.0.98.1:36813): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0bbd4824bea042fd9d14b6c157f6cde1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36813: connect: Connection refused (error 111)
I20260929 01:02:38.866006 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 7617
I20260929 01:02:38.871646 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 6975
I20260929 01:02:38.877477 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 7075
2026-09-29T01:02:38Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5239 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:02:38 dist-test-slave-7zdl krb5kdc[7869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:02:38 dist-test-slave-7zdl krb5kdc[7869](info): set up 2 sockets
Sep 29 01:02:38 dist-test-slave-7zdl krb5kdc[7869](info): commencing operation
krb5kdc: starting...
W20260929 01:02:39.911528 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:02:39 dist-test-slave-7zdl krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:02:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:02:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260929 01:02:40.030119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42293
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:40.138087 7882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:40.138368 7882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:40.138409 7882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:40.142266 7882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:02:40.142318 7882 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 01:02:40.142339 7882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:40.142357 7882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:40.142373 7882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:02:40.142388 7882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:02:40.147409 7882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42293
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:40.148301 7882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:40.149195 7882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:02:40.157568 7882 server_base.cc:1034] running on GCE node
W20260929 01:02:40.157490 7890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:40.157485 7887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:40.157501 7888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:40.158180 7882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:40.159325 7882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:40.160511 7882 hybrid_clock.cc:648] HybridClock initialized: now 1790643760160495 us; error 33 us; skew 500 ppm
Sep 29 01:02:40 dist-test-slave-7zdl krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643760, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:40.163075 7882 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260929 01:02:40.163902 7882 webserver.cc:460] Webserver started at http://127.0.98.62:40255/ using document root <none> and password file <none>
I20260929 01:02:40.164464 7882 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:40.164554 7882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:40.164783 7882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:40.166841 7882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "ad71ec649f3e43ba92633d3a25a3e89c"
format_stamp: "Formatted at 2026-09-29 01:02:40 on dist-test-slave-7zdl"
server_key: "d828cca6f8e11d796a3d9abe36e2b546"
server_key_iv: "9396ec9524f5b28db0636bb2a177ee9a"
server_key_version: "encryptionkey@0"
I20260929 01:02:40.167402 7882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "ad71ec649f3e43ba92633d3a25a3e89c"
format_stamp: "Formatted at 2026-09-29 01:02:40 on dist-test-slave-7zdl"
server_key: "d828cca6f8e11d796a3d9abe36e2b546"
server_key_iv: "9396ec9524f5b28db0636bb2a177ee9a"
server_key_version: "encryptionkey@0"
I20260929 01:02:40.170928 7882 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:02:40.173132 7897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:40.174309 7882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:40.174463 7882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "ad71ec649f3e43ba92633d3a25a3e89c"
format_stamp: "Formatted at 2026-09-29 01:02:40 on dist-test-slave-7zdl"
server_key: "d828cca6f8e11d796a3d9abe36e2b546"
server_key_iv: "9396ec9524f5b28db0636bb2a177ee9a"
server_key_version: "encryptionkey@0"
I20260929 01:02:40.174571 7882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:40.185838 7882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:40.188584 7882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:40.188787 7882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:40.196540 7882 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42293
I20260929 01:02:40.196551 7959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42293 every 8 connection(s)
I20260929 01:02:40.197460 7882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:02:40.200158 7960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:40.205396 7960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c: Bootstrap starting.
I20260929 01:02:40.205722 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 7882
I20260929 01:02:40.205852 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:02:40.206115 392 external_mini_cluster.cc:1428] Setting key f202e68cd2cb37534017b0941cc89f6c
I20260929 01:02:40.207957 7960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:40.208662 7960 log.cc:824] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:40.210480 7960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c: No bootstrap required, opened a new log
I20260929 01:02:40.213943 7960 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad71ec649f3e43ba92633d3a25a3e89c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42293 } }
I20260929 01:02:40.214167 7960 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:40.214247 7960 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad71ec649f3e43ba92633d3a25a3e89c, State: Initialized, Role: FOLLOWER
Sep 29 01:02:40 dist-test-slave-7zdl krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 01:02:40.214700 7960 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad71ec649f3e43ba92633d3a25a3e89c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42293 } }
I20260929 01:02:40.214870 7960 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:02:40.214941 7960 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:02:40.215032 7960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:40.215948 7960 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad71ec649f3e43ba92633d3a25a3e89c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42293 } }
I20260929 01:02:40.216312 7960 leader_election.cc:302] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad71ec649f3e43ba92633d3a25a3e89c; no voters:
I20260929 01:02:40.216657 7960 leader_election.cc:288] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:02:40.216718 7965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:40.216920 7965 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c [term 1 LEADER]: Becoming Leader. State: Replica: ad71ec649f3e43ba92633d3a25a3e89c, State: Running, Role: LEADER
I20260929 01:02:40.217190 7965 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad71ec649f3e43ba92633d3a25a3e89c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42293 } }
I20260929 01:02:40.217911 7960 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:02:40.218919 7966 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad71ec649f3e43ba92633d3a25a3e89c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad71ec649f3e43ba92633d3a25a3e89c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42293 } } }
I20260929 01:02:40.219310 7966 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:40.219210 7967 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad71ec649f3e43ba92633d3a25a3e89c. Latest consensus state: current_term: 1 leader_uuid: "ad71ec649f3e43ba92633d3a25a3e89c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad71ec649f3e43ba92633d3a25a3e89c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42293 } } }
I20260929 01:02:40.219848 7967 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:40.220629 7975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:02:40.221163 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:40.207452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57724 (local address 127.0.98.62:42293)
0929 01:02:40.207876 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:40.207890 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:40.207924 (+ 34us) server_negotiation.cc:407] Connection header received
0929 01:02:40.208608 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:40.208626 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:40.209103 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:40.209468 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:40.210259 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.211317 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.212152 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.212612 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.212781 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:40.215380 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:40.215385 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:40.215398 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:40.215422 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:40.217433 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:40.217883 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:40.217889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:40.217895 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:40.218085 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:40.218413 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:40.218416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:40.218418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:40.218977 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:40.219173 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:40.219925 (+ 752us) server_negotiation.cc:299] Negotiation successful
0929 01:02:40.220209 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":132,"threads_started":1}
I20260929 01:02:40.223662 7975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:02:40.229074 7975 catalog_manager.cc:1347] Generated new cluster ID: f92749751f0b48d78dfabe177f179ca2
I20260929 01:02:40.229137 7975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:02:40.248170 7975 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:02:40.249159 7975 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:02:40.257546 7975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c: Generated new TSK 0
I20260929 01:02:40.258265 7975 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260929 01:02:40.295874 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42293
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:40.409502 7986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:40.409791 7986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:40.409862 7986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:40.413913 7986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:40.413995 7986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:40.414062 7986 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:02:40.414141 7986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:02:40.419467 7986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:40.420524 7986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:40.421511 7986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:40.430678 7994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:40.430755 7986 server_base.cc:1034] running on GCE node
W20260929 01:02:40.430768 7991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:40.430979 7992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:40.431352 7986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:40.431955 7986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:40.433171 7986 hybrid_clock.cc:648] HybridClock initialized: now 1790643760433157 us; error 26 us; skew 500 ppm
Sep 29 01:02:40 dist-test-slave-7zdl krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643760, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:40.435818 7986 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 01:02:40.436630 7986 webserver.cc:460] Webserver started at http://127.0.98.1:37311/ using document root <none> and password file <none>
I20260929 01:02:40.437175 7986 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:40.437264 7986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:40.437511 7986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:40.439723 7986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "48b0eb54a51d43078a7747d7edf55c8e"
format_stamp: "Formatted at 2026-09-29 01:02:40 on dist-test-slave-7zdl"
server_key: "c560ba0f19982becca43ccf56d60bc7d"
server_key_iv: "4aad861079721ad43a34bd9136d9d346"
server_key_version: "encryptionkey@0"
I20260929 01:02:40.440392 7986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "48b0eb54a51d43078a7747d7edf55c8e"
format_stamp: "Formatted at 2026-09-29 01:02:40 on dist-test-slave-7zdl"
server_key: "c560ba0f19982becca43ccf56d60bc7d"
server_key_iv: "4aad861079721ad43a34bd9136d9d346"
server_key_version: "encryptionkey@0"
I20260929 01:02:40.444398 7986 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:02:40.447376 8001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:40.448582 7986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:02:40.448731 7986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "48b0eb54a51d43078a7747d7edf55c8e"
format_stamp: "Formatted at 2026-09-29 01:02:40 on dist-test-slave-7zdl"
server_key: "c560ba0f19982becca43ccf56d60bc7d"
server_key_iv: "4aad861079721ad43a34bd9136d9d346"
server_key_version: "encryptionkey@0"
I20260929 01:02:40.448843 7986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:40.486465 7986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:40.489449 7986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:40.489655 7986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:40.491190 7986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:40.491263 7986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:40.491334 7986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:40.491375 7986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:02:40 dist-test-slave-7zdl krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643760, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 01:02:40.502213 7986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38557
I20260929 01:02:40.502245 8116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38557 every 8 connection(s)
I20260929 01:02:40.503448 7986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:02:40.505546 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:40.491886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48665 (local address 127.0.98.62:42293)
0929 01:02:40.492009 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:40.492012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:40.493018 (+ 1006us) server_negotiation.cc:407] Connection header received
0929 01:02:40.494116 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:40.494119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:40.494183 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:40.494242 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:40.495890 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.496780 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.498130 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.498552 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.498678 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:40.501555 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:40.501558 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:40.501560 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:40.501584 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:40.503307 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:40.503888 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:40.503892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:40.503895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:40.504000 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:40.504452 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:40.504457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:40.504459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:40.504646 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:40.504756 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:40.505240 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 01:02:40.505364 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 01:02:40.506255 8010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:40.492404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42293 (local address 127.0.98.1:48665)
0929 01:02:40.492879 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:40.492920 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:40.493872 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:40.494678 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:40.494689 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:40.495155 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:40.495714 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:40.495730 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.496915 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.496920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:40.497998 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:40.498007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.498732 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.498738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:40.498959 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:40.499556 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:40.499571 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:40.501394 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:40.503415 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:40.503420 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:40.503430 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:40.503755 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:40.504119 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:40.504125 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:40.504129 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:40.504326 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:40.504754 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:40.504761 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:40.505000 (+ 239us) client_negotiation.cc:769] Sending connection context
0929 01:02:40.505184 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 01:02:40.505488 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":135,"threads_started":1}
I20260929 01:02:40.507386 8117 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42293
I20260929 01:02:40.507622 8117 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:40.508148 8117 heartbeater.cc:511] Master 127.0.98.62:42293 requested a full tablet report, sending...
I20260929 01:02:40.509765 7913 ts_manager.cc:194] Registered new tserver with Master: 48b0eb54a51d43078a7747d7edf55c8e (127.0.98.1:38557)
I20260929 01:02:40.511034 7913 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48665
I20260929 01:02:40.512605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 7986
I20260929 01:02:40.512750 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:02:40.513072 392 external_mini_cluster.cc:1428] Setting key ef4a902533b201c6e069e6df474a9657
I20260929 01:02:40.516752 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:40.509760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45141 (local address 127.0.98.62:42293)
0929 01:02:40.509878 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:40.509883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:40.510052 (+ 169us) server_negotiation.cc:407] Connection header received
0929 01:02:40.510164 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:40.510169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:40.510213 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:40.510293 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:40.510769 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.511375 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.512167 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.512408 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.512508 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:40.513669 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:40.513671 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:40.513673 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:40.513715 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:40.515130 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:40.515623 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:40.515626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:40.515627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:40.515691 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:40.515966 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:40.515969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:40.515970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:40.516109 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:40.516238 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:40.516510 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 01:02:40.516611 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 01:02:40.516762 8123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:40.509682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42293 (local address 127.0.98.1:45141)
0929 01:02:40.509968 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:40.509980 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:40.510091 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:40.510306 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:40.510309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:40.510527 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:40.510667 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:40.510673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.511489 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.511493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:40.512052 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:40.512060 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.512511 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.512514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:40.512596 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:40.513192 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:40.513207 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:40.513513 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:40.515252 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:40.515258 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:40.515264 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:40.515530 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:40.515775 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:40.515781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:40.515786 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:40.515881 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:40.516242 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:40.516246 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:40.516339 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 01:02:40.516451 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 01:02:40.516581 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260929 01:02:40.538103 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42293
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:40.646332 8126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:40.646622 8126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:40.646716 8126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:40.650712 8126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:40.650797 8126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:40.650897 8126 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:02:40.650974 8126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:02:40.656251 8126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:40.657274 8126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:40.658257 8126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:40.667740 8132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:40.667773 8131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:40.667755 8134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:40.668147 8126 server_base.cc:1034] running on GCE node
I20260929 01:02:40.668496 8126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:40.669081 8126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:40.670250 8126 hybrid_clock.cc:648] HybridClock initialized: now 1790643760670219 us; error 41 us; skew 500 ppm
Sep 29 01:02:40 dist-test-slave-7zdl krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643760, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:40.673079 8126 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 01:02:40.673974 8126 webserver.cc:460] Webserver started at http://127.0.98.2:40361/ using document root <none> and password file <none>
I20260929 01:02:40.674518 8126 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:40.674607 8126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:40.674876 8126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:40.676873 8126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "609dfbb102b049f598299362032340ed"
format_stamp: "Formatted at 2026-09-29 01:02:40 on dist-test-slave-7zdl"
server_key: "e92ffc6ec966a1857b6ad332d9a19871"
server_key_iv: "826441ce2424754284be1ffb135089f2"
server_key_version: "encryptionkey@0"
I20260929 01:02:40.677428 8126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "609dfbb102b049f598299362032340ed"
format_stamp: "Formatted at 2026-09-29 01:02:40 on dist-test-slave-7zdl"
server_key: "e92ffc6ec966a1857b6ad332d9a19871"
server_key_iv: "826441ce2424754284be1ffb135089f2"
server_key_version: "encryptionkey@0"
I20260929 01:02:40.682145 8126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 01:02:40.684628 8141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:40.685819 8126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:40.685966 8126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "609dfbb102b049f598299362032340ed"
format_stamp: "Formatted at 2026-09-29 01:02:40 on dist-test-slave-7zdl"
server_key: "e92ffc6ec966a1857b6ad332d9a19871"
server_key_iv: "826441ce2424754284be1ffb135089f2"
server_key_version: "encryptionkey@0"
I20260929 01:02:40.686076 8126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:40.707882 8126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:40.710767 8126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:40.711035 8126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:40.712419 8126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:40.712466 8126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:40.712527 8126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:40.712560 8126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:02:40 dist-test-slave-7zdl krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643760, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 01:02:40.723630 8126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42457
I20260929 01:02:40.723971 8256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42457 every 8 connection(s)
I20260929 01:02:40.724762 8126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:02:40.726568 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:40.713264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49699 (local address 127.0.98.62:42293)
0929 01:02:40.713409 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:40.713413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:40.714450 (+ 1037us) server_negotiation.cc:407] Connection header received
0929 01:02:40.715582 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:40.715586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:40.715645 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:40.715754 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:40.717238 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.718138 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.719740 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.720159 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.720291 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:40.722847 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:40.722851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:40.722854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:40.722882 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:40.724556 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:40.725096 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:40.725100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:40.725102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:40.725198 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:40.725618 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:40.725621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:40.725622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:40.725853 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:40.725936 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:40.726347 (+ 411us) server_negotiation.cc:299] Negotiation successful
0929 01:02:40.726455 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 01:02:40.727418 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:40.713829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42293 (local address 127.0.98.2:49699)
0929 01:02:40.714298 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:40.714339 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:40.715366 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:40.715937 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:40.715948 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:40.716461 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:40.717066 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:40.717083 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.718264 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.718267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:40.719621 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:40.719635 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.720374 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.720379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:40.720554 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:40.721124 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:40.721140 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:40.722665 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:40.724679 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:40.724683 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:40.724693 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:40.725000 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:40.725311 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:40.725314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:40.725317 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:40.725495 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:40.725947 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:40.725952 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:40.726155 (+ 203us) client_negotiation.cc:769] Sending connection context
0929 01:02:40.726302 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 01:02:40.726558 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":141,"threads_started":1}
I20260929 01:02:40.728528 8257 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42293
I20260929 01:02:40.728652 8257 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:40.729166 8257 heartbeater.cc:511] Master 127.0.98.62:42293 requested a full tablet report, sending...
I20260929 01:02:40.730209 7913 ts_manager.cc:194] Registered new tserver with Master: 609dfbb102b049f598299362032340ed (127.0.98.2:42457)
I20260929 01:02:40.731032 7913 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49699
I20260929 01:02:40.734338 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 8126
I20260929 01:02:40.734477 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:02:40.734771 392 external_mini_cluster.cc:1428] Setting key c305d644e34c8baf5140f918f38bb25b
I20260929 01:02:40.737496 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:40.730677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34941 (local address 127.0.98.62:42293)
0929 01:02:40.730782 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:40.730786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:40.730981 (+ 195us) server_negotiation.cc:407] Connection header received
0929 01:02:40.731084 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:40.731086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:40.731120 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:40.731210 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:40.731640 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.732109 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.733104 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.733298 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.733374 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:40.734529 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:40.734535 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:40.734540 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:40.734563 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:40.735977 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:40.736405 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:40.736407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:40.736409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:40.736474 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:40.736751 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:40.736753 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:40.736755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:40.736936 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:40.737009 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:40.737290 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 01:02:40.737392 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 01:02:40.737807 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:40.730596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42293 (local address 127.0.98.2:34941)
0929 01:02:40.730902 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:40.730915 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:40.731006 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:40.731225 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:40.731228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:40.731417 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:40.731504 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:40.731509 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.732233 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.732236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:40.733007 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:40.733015 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.733395 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.733397 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:40.733459 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:40.733956 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:40.733967 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:40.734351 (+ 384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:40.736058 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:40.736061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:40.736064 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:40.736293 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:40.736561 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:40.736566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:40.736571 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:40.736664 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:40.737023 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:40.737028 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:40.737137 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 01:02:40.737248 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 01:02:40.737413 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 01:02:40.761540 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42293
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:40.875296 8266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:40.875543 8266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:40.875643 8266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:40.879874 8266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:40.879935 8266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:40.880002 8266 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 01:02:40.880084 8266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:02:40.885315 8266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:40.886415 8266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:40.887499 8266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:40.897289 8271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:40.897316 8272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:40.897316 8274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:40.897558 8266 server_base.cc:1034] running on GCE node
I20260929 01:02:40.898036 8266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:40.898658 8266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:40.899871 8266 hybrid_clock.cc:648] HybridClock initialized: now 1790643760899835 us; error 47 us; skew 500 ppm
Sep 29 01:02:40 dist-test-slave-7zdl krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643760, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:40.902699 8266 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 01:02:40.903649 8266 webserver.cc:460] Webserver started at http://127.0.98.3:36021/ using document root <none> and password file <none>
I20260929 01:02:40.904215 8266 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:40.904361 8266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:40.904595 8266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:40.906740 8266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "41ef31ce05104786bad477cb8fd4bbd5"
format_stamp: "Formatted at 2026-09-29 01:02:40 on dist-test-slave-7zdl"
server_key: "d6e9b1b70b5158387855521c0be4257c"
server_key_iv: "beec2803712fac8cfb1c1b57fd67ed31"
server_key_version: "encryptionkey@0"
I20260929 01:02:40.907420 8266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "41ef31ce05104786bad477cb8fd4bbd5"
format_stamp: "Formatted at 2026-09-29 01:02:40 on dist-test-slave-7zdl"
server_key: "d6e9b1b70b5158387855521c0be4257c"
server_key_iv: "beec2803712fac8cfb1c1b57fd67ed31"
server_key_version: "encryptionkey@0"
I20260929 01:02:40.911197 8266 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:02:40.913601 8281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:40.914917 8266 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:02:40.915047 8266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "41ef31ce05104786bad477cb8fd4bbd5"
format_stamp: "Formatted at 2026-09-29 01:02:40 on dist-test-slave-7zdl"
server_key: "d6e9b1b70b5158387855521c0be4257c"
server_key_iv: "beec2803712fac8cfb1c1b57fd67ed31"
server_key_version: "encryptionkey@0"
I20260929 01:02:40.915153 8266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:40.947376 8266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:40.950670 8266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:40.950902 8266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:40.952487 8266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:40.952557 8266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:40.952625 8266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:40.952679 8266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 01:02:40 dist-test-slave-7zdl krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643760, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 01:02:40.963397 8266 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33501
I20260929 01:02:40.964136 8396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33501 every 8 connection(s)
I20260929 01:02:40.964270 8266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:02:40.966009 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:40.953235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44275 (local address 127.0.98.62:42293)
0929 01:02:40.953402 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:40.953407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:40.954281 (+ 874us) server_negotiation.cc:407] Connection header received
0929 01:02:40.955359 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:40.955364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:40.955430 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:40.955534 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:40.956705 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.957541 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.958937 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.959339 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.959474 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:40.962126 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:40.962133 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:40.962139 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:40.962171 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:40.963922 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:40.964513 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:40.964517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:40.964519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:40.964600 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:40.964998 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:40.965002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:40.965005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:40.965184 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:40.965291 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:40.965734 (+ 443us) server_negotiation.cc:299] Negotiation successful
0929 01:02:40.965845 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 01:02:40.966739 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:40.953678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42293 (local address 127.0.98.3:44275)
0929 01:02:40.954119 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:40.954165 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:40.955098 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:40.955746 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:40.955752 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:40.956110 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:40.956566 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:40.956578 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.957687 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.957691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:40.958766 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:40.958775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.959504 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.959510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:40.959687 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:40.960553 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:40.960568 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:40.961953 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:40.964046 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:40.964053 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:40.964068 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:40.964405 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:40.964696 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:40.964699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:40.964701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:40.964880 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:40.965286 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:40.965292 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:40.965522 (+ 230us) client_negotiation.cc:769] Sending connection context
0929 01:02:40.965693 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 01:02:40.965978 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":141,"threads_started":1}
I20260929 01:02:40.967521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 8266
I20260929 01:02:40.967665 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:02:40.967993 392 external_mini_cluster.cc:1428] Setting key fcc39b9d217b7212527f783621ce0f56
I20260929 01:02:40.968075 8397 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42293
I20260929 01:02:40.968216 8397 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:40.968755 8397 heartbeater.cc:511] Master 127.0.98.62:42293 requested a full tablet report, sending...
I20260929 01:02:40.969753 7913 ts_manager.cc:194] Registered new tserver with Master: 41ef31ce05104786bad477cb8fd4bbd5 (127.0.98.3:33501)
I20260929 01:02:40.970518 7913 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44275
I20260929 01:02:40.976785 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:40.970229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42187 (local address 127.0.98.62:42293)
0929 01:02:40.970339 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:40.970342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:40.970513 (+ 171us) server_negotiation.cc:407] Connection header received
0929 01:02:40.970634 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:40.970636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:40.970668 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:40.970760 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:40.971193 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.971681 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.972470 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.972694 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.972796 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:40.973790 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:40.973794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:40.973796 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:40.973818 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:40.975256 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:40.975694 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:40.975699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:40.975701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:40.975775 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:40.976042 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:40.976046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:40.976048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:40.976203 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:40.976301 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:40.976552 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 01:02:40.976647 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 01:02:40.976796 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:40.970176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42293 (local address 127.0.98.3:42187)
0929 01:02:40.970450 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:40.970463 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:40.970555 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:40.970821 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:40.970827 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:40.970997 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:40.971084 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:40.971089 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.971797 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.971800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:40.972360 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:40.972367 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:40.972796 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:40.972799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:40.972898 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:40.973365 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:40.973378 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:40.973658 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:40.975389 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:40.975392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:40.975394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:40.975599 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:40.975872 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:40.975875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:40.975877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:40.975957 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:40.976302 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:40.976305 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:40.976407 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 01:02:40.976506 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 01:02:40.976631 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":87,"threads_started":1}
I20260929 01:02:40.981523 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 01:02:40 dist-test-slave-7zdl krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643760, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 01:02:41 dist-test-slave-7zdl krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:41 dist-test-slave-7zdl krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 01:02:41.004537 7963 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:40.998651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57734 (local address 127.0.98.62:42293)
0929 01:02:40.998769 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:40.998773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:40.998784 (+ 11us) server_negotiation.cc:407] Connection header received
0929 01:02:40.998870 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:40.998877 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:40.998916 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:40.998982 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:40.999640 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.000135 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.000976 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.001171 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.001241 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:41.004349 (+ 3108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57734: BlockingRecv error: recv got EOF from 127.0.0.1:57734 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 01:02:41 dist-test-slave-7zdl krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643760, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 01:02:41.013248 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:41.005849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57738 (local address 127.0.98.62:42293)
0929 01:02:41.005977 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:41.005981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:41.006081 (+ 100us) server_negotiation.cc:407] Connection header received
0929 01:02:41.006168 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:41.006170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:41.006202 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:41.006265 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:41.006787 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.007316 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.008123 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.008315 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.008400 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:41.010498 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:41.010501 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:41.010503 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:41.010523 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:41.011985 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:41.012323 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:41.012326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:41.012327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:41.012404 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:41.012619 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:41.012621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:41.012622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:41.012786 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:41.012883 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:41.013063 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 01:02:41.013144 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 01:02:41.015864 7913 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:02:41.019027 7913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:02:41.032421 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:41.026683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33501 (local address 127.0.0.1:57930)
0929 01:02:41.027834 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:41.027849 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:41.028236 (+ 387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:41.028706 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:41.028711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:41.028732 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:41.028964 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:41.028970 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.029930 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.029936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:41.031102 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:41.031110 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.031888 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.031891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:41.031994 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:41.032033 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 01:02:41.032143 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 01:02:41.032252 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":59,"spinlock_wait_cycles":409856,"thread_start_us":65,"threads_started":1}
I20260929 01:02:41.032426 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:41.026848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57930 (local address 127.0.98.3:33501)
0929 01:02:41.027267 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:41.027279 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:41.027956 (+ 677us) server_negotiation.cc:407] Connection header received
0929 01:02:41.028403 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:41.028408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:41.028565 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:41.028716 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:41.029089 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.029817 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.031212 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.031763 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.031882 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:41.032041 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:41.032170 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 01:02:41.032260 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":106,"threads_started":1}
I20260929 01:02:41.033528 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:41.026956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42457 (local address 127.0.0.1:48188)
0929 01:02:41.027735 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:41.027786 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:41.027981 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:41.028720 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:41.028723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:41.028753 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:41.028971 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:41.028977 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.030412 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.030415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:41.031804 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:41.031811 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.033039 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.033042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:41.033165 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:41.033218 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 01:02:41.033322 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 01:02:41.033369 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":374,"spinlock_wait_cycles":4096,"thread_start_us":80,"threads_started":1}
I20260929 01:02:41.033721 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:41.027138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48188 (local address 127.0.98.2:42457)
0929 01:02:41.027449 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:41.027462 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:41.027880 (+ 418us) server_negotiation.cc:407] Connection header received
0929 01:02:41.028171 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:41.028177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:41.028393 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:41.028541 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:41.029098 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.030270 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.031918 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.032718 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.032808 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:41.032957 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:41.033382 (+ 425us) server_negotiation.cc:299] Negotiation successful
0929 01:02:41.033479 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":108,"threads_started":1}
I20260929 01:02:41.033741 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:41.027418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38557 (local address 127.0.0.1:56970)
0929 01:02:41.027888 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:41.027902 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:41.028206 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:41.028720 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:41.028723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:41.028751 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:41.028962 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:41.028969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.030412 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.030415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:41.031677 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:41.031687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.033015 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.033020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:41.033158 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:41.033221 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 01:02:41.033289 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 01:02:41.033360 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":96,"spinlock_wait_cycles":5376,"thread_start_us":84,"threads_started":1}
I20260929 01:02:41.033936 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:41.027551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56970 (local address 127.0.98.1:38557)
0929 01:02:41.028131 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:41.028137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:41.028155 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:02:41.028327 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:41.028332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:41.028482 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:41.028599 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:41.029131 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.030294 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.031809 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.032736 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.032824 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:41.032951 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:41.033480 (+ 529us) server_negotiation.cc:299] Negotiation successful
0929 01:02:41.033668 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":84,"threads_started":1}
I20260929 01:02:41.035311 8330 tablet_service.cc:1467] Processing CreateTablet for tablet c2de1930df6b4b8ab91778783c4183fb (DEFAULT_TABLE table=test-table [id=95a14f4393c741ee8173e04ca11c8aa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:41.036423 8330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2de1930df6b4b8ab91778783c4183fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:41.036326 8191 tablet_service.cc:1467] Processing CreateTablet for tablet c2de1930df6b4b8ab91778783c4183fb (DEFAULT_TABLE table=test-table [id=95a14f4393c741ee8173e04ca11c8aa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:41.036468 8051 tablet_service.cc:1467] Processing CreateTablet for tablet c2de1930df6b4b8ab91778783c4183fb (DEFAULT_TABLE table=test-table [id=95a14f4393c741ee8173e04ca11c8aa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:41.037194 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2de1930df6b4b8ab91778783c4183fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:41.037391 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2de1930df6b4b8ab91778783c4183fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:41.044003 8426 tablet_bootstrap.cc:492] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e: Bootstrap starting.
I20260929 01:02:41.044667 8427 tablet_bootstrap.cc:492] T c2de1930df6b4b8ab91778783c4183fb P 41ef31ce05104786bad477cb8fd4bbd5: Bootstrap starting.
I20260929 01:02:41.044730 8425 tablet_bootstrap.cc:492] T c2de1930df6b4b8ab91778783c4183fb P 609dfbb102b049f598299362032340ed: Bootstrap starting.
I20260929 01:02:41.046253 8427 tablet_bootstrap.cc:654] T c2de1930df6b4b8ab91778783c4183fb P 41ef31ce05104786bad477cb8fd4bbd5: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:41.046357 8426 tablet_bootstrap.cc:654] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:41.046492 8425 tablet_bootstrap.cc:654] T c2de1930df6b4b8ab91778783c4183fb P 609dfbb102b049f598299362032340ed: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:41.047097 8427 log.cc:824] T c2de1930df6b4b8ab91778783c4183fb P 41ef31ce05104786bad477cb8fd4bbd5: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:41.047214 8425 log.cc:824] T c2de1930df6b4b8ab91778783c4183fb P 609dfbb102b049f598299362032340ed: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:41.047214 8426 log.cc:824] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:41.048924 8427 tablet_bootstrap.cc:492] T c2de1930df6b4b8ab91778783c4183fb P 41ef31ce05104786bad477cb8fd4bbd5: No bootstrap required, opened a new log
I20260929 01:02:41.048915 8426 tablet_bootstrap.cc:492] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e: No bootstrap required, opened a new log
I20260929 01:02:41.049115 8427 ts_tablet_manager.cc:1397] T c2de1930df6b4b8ab91778783c4183fb P 41ef31ce05104786bad477cb8fd4bbd5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 01:02:41.049127 8426 ts_tablet_manager.cc:1397] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 01:02:41.051841 8427 raft_consensus.cc:348] T c2de1930df6b4b8ab91778783c4183fb P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } } peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } }
I20260929 01:02:41.052069 8427 raft_consensus.cc:374] T c2de1930df6b4b8ab91778783c4183fb P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:41.052131 8427 raft_consensus.cc:729] T c2de1930df6b4b8ab91778783c4183fb P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41ef31ce05104786bad477cb8fd4bbd5, State: Initialized, Role: FOLLOWER
I20260929 01:02:41.052670 8427 consensus_queue.cc:260] T c2de1930df6b4b8ab91778783c4183fb P 41ef31ce05104786bad477cb8fd4bbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } } peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } }
I20260929 01:02:41.052591 8426 raft_consensus.cc:348] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } } peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } }
I20260929 01:02:41.052812 8426 raft_consensus.cc:374] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:41.052873 8426 raft_consensus.cc:729] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b0eb54a51d43078a7747d7edf55c8e, State: Initialized, Role: FOLLOWER
I20260929 01:02:41.053290 8397 heartbeater.cc:503] Master 127.0.98.62:42293 was elected leader, sending a full tablet report...
I20260929 01:02:41.053483 8425 tablet_bootstrap.cc:492] T c2de1930df6b4b8ab91778783c4183fb P 609dfbb102b049f598299362032340ed: No bootstrap required, opened a new log
I20260929 01:02:41.053668 8425 ts_tablet_manager.cc:1397] T c2de1930df6b4b8ab91778783c4183fb P 609dfbb102b049f598299362032340ed: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260929 01:02:41.053695 8427 ts_tablet_manager.cc:1428] T c2de1930df6b4b8ab91778783c4183fb P 41ef31ce05104786bad477cb8fd4bbd5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 01:02:41.054394 8426 consensus_queue.cc:260] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } } peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } }
I20260929 01:02:41.055012 8117 heartbeater.cc:503] Master 127.0.98.62:42293 was elected leader, sending a full tablet report...
I20260929 01:02:41.055387 8426 ts_tablet_manager.cc:1428] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 01:02:41.057365 8425 raft_consensus.cc:348] T c2de1930df6b4b8ab91778783c4183fb P 609dfbb102b049f598299362032340ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } } peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } }
I20260929 01:02:41.057644 8425 raft_consensus.cc:374] T c2de1930df6b4b8ab91778783c4183fb P 609dfbb102b049f598299362032340ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:41.057717 8425 raft_consensus.cc:729] T c2de1930df6b4b8ab91778783c4183fb P 609dfbb102b049f598299362032340ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 609dfbb102b049f598299362032340ed, State: Initialized, Role: FOLLOWER
I20260929 01:02:41.058138 8425 consensus_queue.cc:260] T c2de1930df6b4b8ab91778783c4183fb P 609dfbb102b049f598299362032340ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } } peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } }
I20260929 01:02:41.058874 8257 heartbeater.cc:503] Master 127.0.98.62:42293 was elected leader, sending a full tablet report...
I20260929 01:02:41.059109 8425 ts_tablet_manager.cc:1428] T c2de1930df6b4b8ab91778783c4183fb P 609dfbb102b049f598299362032340ed: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260929 01:02:41.215524 8398 tablet.cc:2369] T c2de1930df6b4b8ab91778783c4183fb P 41ef31ce05104786bad477cb8fd4bbd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:02:41.225857 8258 tablet.cc:2369] T c2de1930df6b4b8ab91778783c4183fb P 609dfbb102b049f598299362032340ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:02:41.254750 8118 tablet.cc:2369] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:02:41.309626 8432 raft_consensus.cc:482] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:02:41.309880 8432 raft_consensus.cc:504] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } } peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } }
I20260929 01:02:41.311089 8432 leader_election.cc:288] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41ef31ce05104786bad477cb8fd4bbd5 (127.0.98.3:33501), 609dfbb102b049f598299362032340ed (127.0.98.2:42457)
I20260929 01:02:41.314998 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:41.311459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56473 (local address 127.0.98.2:42457)
0929 01:02:41.311601 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:41.311606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:41.311882 (+ 276us) server_negotiation.cc:407] Connection header received
0929 01:02:41.312028 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:41.312033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:41.312082 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:41.312164 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:41.312489 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.312987 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.314054 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.314435 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.314559 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:41.314619 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:41.314810 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 01:02:41.314883 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 01:02:41.315032 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:41.311341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42457 (local address 127.0.98.1:56473)
0929 01:02:41.311723 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:41.311734 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:41.311885 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:41.312188 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:41.312190 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:41.312205 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:41.312377 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:41.312382 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.313098 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.313102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:41.313932 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:41.313940 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.314562 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.314566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:41.314661 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:41.314682 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 01:02:41.314777 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 01:02:41.314862 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":34,"spinlock_wait_cycles":7680,"thread_start_us":120,"threads_started":1}
I20260929 01:02:41.315043 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:41.311425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44433 (local address 127.0.98.3:33501)
0929 01:02:41.311566 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:41.311570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:41.311835 (+ 265us) server_negotiation.cc:407] Connection header received
0929 01:02:41.312035 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:41.312038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:41.312083 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:41.312166 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:41.312483 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.312986 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.314081 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.314477 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.314592 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:41.314655 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:41.314844 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 01:02:41.314916 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 01:02:41.315259 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:41.311341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33501 (local address 127.0.98.1:44433)
0929 01:02:41.311716 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:41.311728 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:41.311883 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:41.312177 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:41.312179 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:41.312194 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:41.312360 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:41.312366 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.313095 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.313101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:41.313958 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:41.313966 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.314593 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.314597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:41.314699 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:41.314726 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 01:02:41.314846 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 01:02:41.314913 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":60,"spinlock_wait_cycles":25344,"thread_start_us":159,"threads_started":1}
I20260929 01:02:41.315644 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2de1930df6b4b8ab91778783c4183fb" candidate_uuid: "48b0eb54a51d43078a7747d7edf55c8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "609dfbb102b049f598299362032340ed" is_pre_election: true
I20260929 01:02:41.315857 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2de1930df6b4b8ab91778783c4183fb" candidate_uuid: "48b0eb54a51d43078a7747d7edf55c8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41ef31ce05104786bad477cb8fd4bbd5" is_pre_election: true
I20260929 01:02:41.315950 8211 raft_consensus.cc:2454] T c2de1930df6b4b8ab91778783c4183fb P 609dfbb102b049f598299362032340ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48b0eb54a51d43078a7747d7edf55c8e in term 0.
I20260929 01:02:41.316150 8347 raft_consensus.cc:2454] T c2de1930df6b4b8ab91778783c4183fb P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48b0eb54a51d43078a7747d7edf55c8e in term 0.
I20260929 01:02:41.316473 8004 leader_election.cc:302] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48b0eb54a51d43078a7747d7edf55c8e, 609dfbb102b049f598299362032340ed; no voters:
I20260929 01:02:41.316756 8432 raft_consensus.cc:2781] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:02:41.316836 8432 raft_consensus.cc:482] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:02:41.316895 8432 raft_consensus.cc:3037] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:41.317919 8432 raft_consensus.cc:504] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } } peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } }
I20260929 01:02:41.318318 8432 leader_election.cc:288] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e [CANDIDATE]: Term 1 election: Requested vote from peers 41ef31ce05104786bad477cb8fd4bbd5 (127.0.98.3:33501), 609dfbb102b049f598299362032340ed (127.0.98.2:42457)
I20260929 01:02:41.318602 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2de1930df6b4b8ab91778783c4183fb" candidate_uuid: "48b0eb54a51d43078a7747d7edf55c8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41ef31ce05104786bad477cb8fd4bbd5"
I20260929 01:02:41.318640 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2de1930df6b4b8ab91778783c4183fb" candidate_uuid: "48b0eb54a51d43078a7747d7edf55c8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "609dfbb102b049f598299362032340ed"
I20260929 01:02:41.318742 8347 raft_consensus.cc:3037] T c2de1930df6b4b8ab91778783c4183fb P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:41.318782 8211 raft_consensus.cc:3037] T c2de1930df6b4b8ab91778783c4183fb P 609dfbb102b049f598299362032340ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:41.319918 8211 raft_consensus.cc:2454] T c2de1930df6b4b8ab91778783c4183fb P 609dfbb102b049f598299362032340ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48b0eb54a51d43078a7747d7edf55c8e in term 1.
I20260929 01:02:41.320191 8004 leader_election.cc:302] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48b0eb54a51d43078a7747d7edf55c8e, 609dfbb102b049f598299362032340ed; no voters:
I20260929 01:02:41.320343 8432 raft_consensus.cc:2781] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:41.320480 8347 raft_consensus.cc:2454] T c2de1930df6b4b8ab91778783c4183fb P 41ef31ce05104786bad477cb8fd4bbd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48b0eb54a51d43078a7747d7edf55c8e in term 1.
I20260929 01:02:41.320590 8432 raft_consensus.cc:686] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e [term 1 LEADER]: Becoming Leader. State: Replica: 48b0eb54a51d43078a7747d7edf55c8e, State: Running, Role: LEADER
I20260929 01:02:41.320928 8432 consensus_queue.cc:237] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } } peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } }
I20260929 01:02:41.324059 7914 catalog_manager.cc:5495] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 48b0eb54a51d43078a7747d7edf55c8e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "48b0eb54a51d43078a7747d7edf55c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } health_report { overall_health: UNKNOWN } } }
I20260929 01:02:41.399854 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:41.396778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56812 (local address 127.0.98.1:38557)
0929 01:02:41.396897 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:41.396901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:41.396914 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:02:41.397001 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:41.397003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:41.397060 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:41.397147 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:02:41.397345 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.397829 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.398598 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.398853 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.398956 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:41.399149 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:02:41.399568 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:02:41.399649 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:41.399705 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 01:02:41.399751 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 01:02:41.405471 8211 raft_consensus.cc:1261] T c2de1930df6b4b8ab91778783c4183fb P 609dfbb102b049f598299362032340ed [term 1 FOLLOWER]: Refusing update from remote peer 48b0eb54a51d43078a7747d7edf55c8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:41.405508 8347 raft_consensus.cc:1261] T c2de1930df6b4b8ab91778783c4183fb P 41ef31ce05104786bad477cb8fd4bbd5 [term 1 FOLLOWER]: Refusing update from remote peer 48b0eb54a51d43078a7747d7edf55c8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:41.406185 8432 consensus_queue.cc:1035] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:41.406388 8436 consensus_queue.cc:1035] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:41.417580 8440 mvcc.cc:205] Tried to move back new op lower bound from 7334476846709690368 to 7334476846373285888. Current Snapshot: MvccSnapshot[applied={T|T < 7334476846709690368 or (T in {7334476846709690368})}]
I20260929 01:02:41.421798 7913 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:41.421940 7913 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:41.424211 7913 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c: Sending DeleteTablet for 3 replicas of tablet c2de1930df6b4b8ab91778783c4183fb
I20260929 01:02:41.424999 8330 tablet_service.cc:1514] Processing DeleteTablet for tablet c2de1930df6b4b8ab91778783c4183fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:41 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57930
I20260929 01:02:41.425009 8051 tablet_service.cc:1514] Processing DeleteTablet for tablet c2de1930df6b4b8ab91778783c4183fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:41 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56970
I20260929 01:02:41.425472 8449 tablet_replica.cc:330] T c2de1930df6b4b8ab91778783c4183fb P 41ef31ce05104786bad477cb8fd4bbd5: stopping tablet replica
I20260929 01:02:41.425477 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet c2de1930df6b4b8ab91778783c4183fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:41 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48188
I20260929 01:02:41.425768 8448 tablet_replica.cc:330] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e: stopping tablet replica
I20260929 01:02:41.425860 8450 tablet_replica.cc:330] T c2de1930df6b4b8ab91778783c4183fb P 609dfbb102b049f598299362032340ed: stopping tablet replica
I20260929 01:02:41.426126 8450 raft_consensus.cc:2229] T c2de1930df6b4b8ab91778783c4183fb P 609dfbb102b049f598299362032340ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:41.426126 8449 raft_consensus.cc:2229] T c2de1930df6b4b8ab91778783c4183fb P 41ef31ce05104786bad477cb8fd4bbd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:41.426211 7913 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 01:02:41.426427 8450 raft_consensus.cc:2258] T c2de1930df6b4b8ab91778783c4183fb P 609dfbb102b049f598299362032340ed [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 01:02:41.426604 7913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:02:41.426652 8448 raft_consensus.cc:2229] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:02:41.427042 8449 raft_consensus.cc:2258] T c2de1930df6b4b8ab91778783c4183fb P 41ef31ce05104786bad477cb8fd4bbd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:41.427421 8448 raft_consensus.cc:2258] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:41.427872 8450 ts_tablet_manager.cc:1905] T c2de1930df6b4b8ab91778783c4183fb P 609dfbb102b049f598299362032340ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:41.429016 8449 ts_tablet_manager.cc:1905] T c2de1930df6b4b8ab91778783c4183fb P 41ef31ce05104786bad477cb8fd4bbd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:41.429855 8448 ts_tablet_manager.cc:1905] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:41.430908 8450 ts_tablet_manager.cc:1918] T c2de1930df6b4b8ab91778783c4183fb P 609dfbb102b049f598299362032340ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:02:41.431010 8450 log.cc:1196] T c2de1930df6b4b8ab91778783c4183fb P 609dfbb102b049f598299362032340ed: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-1/wal/wals/c2de1930df6b4b8ab91778783c4183fb
I20260929 01:02:41.431301 8450 ts_tablet_manager.cc:1939] T c2de1930df6b4b8ab91778783c4183fb P 609dfbb102b049f598299362032340ed: Deleting consensus metadata
I20260929 01:02:41.432000 8449 ts_tablet_manager.cc:1918] T c2de1930df6b4b8ab91778783c4183fb P 41ef31ce05104786bad477cb8fd4bbd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:02:41.432031 8330 tablet_service.cc:1467] Processing CreateTablet for tablet 580b6f5520c34d9d8a2e04cd7998e349 (DEFAULT_TABLE table=test-table [id=2afee3385a2c4226b8510d43399f1447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:41.432185 8449 log.cc:1196] T c2de1930df6b4b8ab91778783c4183fb P 41ef31ce05104786bad477cb8fd4bbd5: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/c2de1930df6b4b8ab91778783c4183fb
I20260929 01:02:41.432307 8330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 580b6f5520c34d9d8a2e04cd7998e349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:41.432452 8448 ts_tablet_manager.cc:1918] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:02:41.432546 8448 log.cc:1196] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-0/wal/wals/c2de1930df6b4b8ab91778783c4183fb
I20260929 01:02:41.432634 8449 ts_tablet_manager.cc:1939] T c2de1930df6b4b8ab91778783c4183fb P 41ef31ce05104786bad477cb8fd4bbd5: Deleting consensus metadata
I20260929 01:02:41.432852 8448 ts_tablet_manager.cc:1939] T c2de1930df6b4b8ab91778783c4183fb P 48b0eb54a51d43078a7747d7edf55c8e: Deleting consensus metadata
I20260929 01:02:41.433635 7899 catalog_manager.cc:4841] TS 41ef31ce05104786bad477cb8fd4bbd5 (127.0.98.3:33501): tablet c2de1930df6b4b8ab91778783c4183fb (table test-table [id=95a14f4393c741ee8173e04ca11c8aa8]) successfully deleted
I20260929 01:02:41.434535 8427 tablet_bootstrap.cc:492] T 580b6f5520c34d9d8a2e04cd7998e349 P 41ef31ce05104786bad477cb8fd4bbd5: Bootstrap starting.
I20260929 01:02:41.434706 8191 tablet_service.cc:1467] Processing CreateTablet for tablet 580b6f5520c34d9d8a2e04cd7998e349 (DEFAULT_TABLE table=test-table [id=2afee3385a2c4226b8510d43399f1447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:41.434971 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 580b6f5520c34d9d8a2e04cd7998e349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:41.435365 8427 tablet_bootstrap.cc:654] T 580b6f5520c34d9d8a2e04cd7998e349 P 41ef31ce05104786bad477cb8fd4bbd5: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:41.436311 8427 tablet_bootstrap.cc:492] T 580b6f5520c34d9d8a2e04cd7998e349 P 41ef31ce05104786bad477cb8fd4bbd5: No bootstrap required, opened a new log
I20260929 01:02:41.436419 8427 ts_tablet_manager.cc:1397] T 580b6f5520c34d9d8a2e04cd7998e349 P 41ef31ce05104786bad477cb8fd4bbd5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 01:02:41.436800 8427 raft_consensus.cc:348] T 580b6f5520c34d9d8a2e04cd7998e349 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:41.436916 8427 raft_consensus.cc:374] T 580b6f5520c34d9d8a2e04cd7998e349 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:41.436956 8427 raft_consensus.cc:729] T 580b6f5520c34d9d8a2e04cd7998e349 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41ef31ce05104786bad477cb8fd4bbd5, State: Initialized, Role: FOLLOWER
I20260929 01:02:41.437098 8425 tablet_bootstrap.cc:492] T 580b6f5520c34d9d8a2e04cd7998e349 P 609dfbb102b049f598299362032340ed: Bootstrap starting.
I20260929 01:02:41.437098 8427 consensus_queue.cc:260] T 580b6f5520c34d9d8a2e04cd7998e349 P 41ef31ce05104786bad477cb8fd4bbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:41.437347 8427 ts_tablet_manager.cc:1428] T 580b6f5520c34d9d8a2e04cd7998e349 P 41ef31ce05104786bad477cb8fd4bbd5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:41.438030 8425 tablet_bootstrap.cc:654] T 580b6f5520c34d9d8a2e04cd7998e349 P 609dfbb102b049f598299362032340ed: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:41.438971 7900 catalog_manager.cc:4841] TS 609dfbb102b049f598299362032340ed (127.0.98.2:42457): tablet c2de1930df6b4b8ab91778783c4183fb (table test-table [id=95a14f4393c741ee8173e04ca11c8aa8]) successfully deleted
I20260929 01:02:41.439322 7899 catalog_manager.cc:4841] TS 48b0eb54a51d43078a7747d7edf55c8e (127.0.98.1:38557): tablet c2de1930df6b4b8ab91778783c4183fb (table test-table [id=95a14f4393c741ee8173e04ca11c8aa8]) successfully deleted
I20260929 01:02:41.439380 8051 tablet_service.cc:1467] Processing CreateTablet for tablet 580b6f5520c34d9d8a2e04cd7998e349 (DEFAULT_TABLE table=test-table [id=2afee3385a2c4226b8510d43399f1447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:41.439620 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 580b6f5520c34d9d8a2e04cd7998e349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:41.439781 8425 tablet_bootstrap.cc:492] T 580b6f5520c34d9d8a2e04cd7998e349 P 609dfbb102b049f598299362032340ed: No bootstrap required, opened a new log
I20260929 01:02:41.439848 8425 ts_tablet_manager.cc:1397] T 580b6f5520c34d9d8a2e04cd7998e349 P 609dfbb102b049f598299362032340ed: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 01:02:41.440250 8425 raft_consensus.cc:348] T 580b6f5520c34d9d8a2e04cd7998e349 P 609dfbb102b049f598299362032340ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:41.440362 8425 raft_consensus.cc:374] T 580b6f5520c34d9d8a2e04cd7998e349 P 609dfbb102b049f598299362032340ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:41.440400 8425 raft_consensus.cc:729] T 580b6f5520c34d9d8a2e04cd7998e349 P 609dfbb102b049f598299362032340ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 609dfbb102b049f598299362032340ed, State: Initialized, Role: FOLLOWER
I20260929 01:02:41.440516 8425 consensus_queue.cc:260] T 580b6f5520c34d9d8a2e04cd7998e349 P 609dfbb102b049f598299362032340ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:41.440749 8425 ts_tablet_manager.cc:1428] T 580b6f5520c34d9d8a2e04cd7998e349 P 609dfbb102b049f598299362032340ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 01:02:41.442039 8426 tablet_bootstrap.cc:492] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e: Bootstrap starting.
I20260929 01:02:41.442761 8426 tablet_bootstrap.cc:654] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:41.443934 8426 tablet_bootstrap.cc:492] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e: No bootstrap required, opened a new log
I20260929 01:02:41.444027 8426 ts_tablet_manager.cc:1397] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 01:02:41.444438 8426 raft_consensus.cc:348] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:41.444545 8426 raft_consensus.cc:374] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:41.444579 8426 raft_consensus.cc:729] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b0eb54a51d43078a7747d7edf55c8e, State: Initialized, Role: FOLLOWER
I20260929 01:02:41.444687 8426 consensus_queue.cc:260] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:41.444911 8426 ts_tablet_manager.cc:1428] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 01:02:41.523751 8436 raft_consensus.cc:482] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:02:41.523892 8436 raft_consensus.cc:504] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:41.524325 8436 leader_election.cc:288] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 609dfbb102b049f598299362032340ed (127.0.98.2:42457), 41ef31ce05104786bad477cb8fd4bbd5 (127.0.98.3:33501)
I20260929 01:02:41.524746 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "580b6f5520c34d9d8a2e04cd7998e349" candidate_uuid: "48b0eb54a51d43078a7747d7edf55c8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "609dfbb102b049f598299362032340ed" is_pre_election: true
I20260929 01:02:41.524752 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "580b6f5520c34d9d8a2e04cd7998e349" candidate_uuid: "48b0eb54a51d43078a7747d7edf55c8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41ef31ce05104786bad477cb8fd4bbd5" is_pre_election: true
I20260929 01:02:41.524917 8347 raft_consensus.cc:2454] T 580b6f5520c34d9d8a2e04cd7998e349 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48b0eb54a51d43078a7747d7edf55c8e in term 0.
I20260929 01:02:41.524919 8211 raft_consensus.cc:2454] T 580b6f5520c34d9d8a2e04cd7998e349 P 609dfbb102b049f598299362032340ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48b0eb54a51d43078a7747d7edf55c8e in term 0.
I20260929 01:02:41.525246 8003 leader_election.cc:302] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41ef31ce05104786bad477cb8fd4bbd5, 48b0eb54a51d43078a7747d7edf55c8e; no voters:
I20260929 01:02:41.525465 8436 raft_consensus.cc:2781] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:02:41.525520 8436 raft_consensus.cc:482] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:02:41.525559 8436 raft_consensus.cc:3037] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:41.527050 8436 raft_consensus.cc:504] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:41.527406 8436 leader_election.cc:288] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e [CANDIDATE]: Term 1 election: Requested vote from peers 609dfbb102b049f598299362032340ed (127.0.98.2:42457), 41ef31ce05104786bad477cb8fd4bbd5 (127.0.98.3:33501)
I20260929 01:02:41.527738 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "580b6f5520c34d9d8a2e04cd7998e349" candidate_uuid: "48b0eb54a51d43078a7747d7edf55c8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41ef31ce05104786bad477cb8fd4bbd5"
I20260929 01:02:41.527802 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "580b6f5520c34d9d8a2e04cd7998e349" candidate_uuid: "48b0eb54a51d43078a7747d7edf55c8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "609dfbb102b049f598299362032340ed"
I20260929 01:02:41.527869 8347 raft_consensus.cc:3037] T 580b6f5520c34d9d8a2e04cd7998e349 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:41.527930 8211 raft_consensus.cc:3037] T 580b6f5520c34d9d8a2e04cd7998e349 P 609dfbb102b049f598299362032340ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:41.528947 8347 raft_consensus.cc:2454] T 580b6f5520c34d9d8a2e04cd7998e349 P 41ef31ce05104786bad477cb8fd4bbd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48b0eb54a51d43078a7747d7edf55c8e in term 1.
I20260929 01:02:41.529150 8211 raft_consensus.cc:2454] T 580b6f5520c34d9d8a2e04cd7998e349 P 609dfbb102b049f598299362032340ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48b0eb54a51d43078a7747d7edf55c8e in term 1.
I20260929 01:02:41.529284 8003 leader_election.cc:302] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41ef31ce05104786bad477cb8fd4bbd5, 48b0eb54a51d43078a7747d7edf55c8e; no voters:
I20260929 01:02:41.529454 8436 raft_consensus.cc:2781] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:41.529546 8436 raft_consensus.cc:686] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e [term 1 LEADER]: Becoming Leader. State: Replica: 48b0eb54a51d43078a7747d7edf55c8e, State: Running, Role: LEADER
I20260929 01:02:41.529685 8436 consensus_queue.cc:237] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:41.531312 7913 catalog_manager.cc:5495] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 48b0eb54a51d43078a7747d7edf55c8e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "48b0eb54a51d43078a7747d7edf55c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } health_report { overall_health: UNKNOWN } } }
Sep 29 01:02:41 dist-test-slave-7zdl krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643760, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 01:02:41.581604 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:41.572442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42293 (local address 127.0.0.1:54416)
0929 01:02:41.572885 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:41.572898 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:41.572996 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:41.573265 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:41.573267 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:41.573650 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:41.573746 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:41.573751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.574903 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.574906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:41.575452 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:41.575458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.576026 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.576028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:41.576087 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:41.576678 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:41.576690 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:41.578343 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:41.580226 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:41.580234 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:41.580239 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:41.580463 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:41.580734 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:41.580736 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:41.580737 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:41.580816 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:41.581175 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:41.581180 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:41.581273 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 01:02:41.581371 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 01:02:41.581472 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":90,"threads_started":1}
I20260929 01:02:41.581727 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:41.572549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.0.98.62:42293)
0929 01:02:41.572821 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:41.572830 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:41.572970 (+ 140us) server_negotiation.cc:407] Connection header received
0929 01:02:41.573082 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:41.573085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:41.573135 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:41.573241 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:41.573865 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.574715 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.575566 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.575909 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.576020 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:41.578474 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:41.578478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:41.578481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:41.578504 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:41.580119 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:41.580560 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:41.580563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:41.580565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:41.580650 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:41.580895 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:41.580898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:41.580900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:41.581088 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:41.581175 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:41.581422 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 01:02:41.581535 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":81,"threads_started":1}
I20260929 01:02:41.585832 7913 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54416:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 01:02:41.586560 7913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:02:41.591073 8051 tablet_service.cc:1467] Processing CreateTablet for tablet 45bb7d5583874a43bb6aa4af4c0bf7c9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50249fd06b644cf4afb400a766b04567]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:02:41.591285 8330 tablet_service.cc:1467] Processing CreateTablet for tablet 45bb7d5583874a43bb6aa4af4c0bf7c9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50249fd06b644cf4afb400a766b04567]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:02:41.591255 8191 tablet_service.cc:1467] Processing CreateTablet for tablet 45bb7d5583874a43bb6aa4af4c0bf7c9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50249fd06b644cf4afb400a766b04567]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 01:02:41.591457 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45bb7d5583874a43bb6aa4af4c0bf7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:41.591605 8330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45bb7d5583874a43bb6aa4af4c0bf7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:41.591604 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45bb7d5583874a43bb6aa4af4c0bf7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:41.593523 8426 tablet_bootstrap.cc:492] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e: Bootstrap starting.
I20260929 01:02:41.594286 8426 tablet_bootstrap.cc:654] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:41.594316 8427 tablet_bootstrap.cc:492] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 41ef31ce05104786bad477cb8fd4bbd5: Bootstrap starting.
I20260929 01:02:41.594946 8425 tablet_bootstrap.cc:492] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 609dfbb102b049f598299362032340ed: Bootstrap starting.
I20260929 01:02:41.595153 8427 tablet_bootstrap.cc:654] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 41ef31ce05104786bad477cb8fd4bbd5: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:41.595799 8425 tablet_bootstrap.cc:654] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 609dfbb102b049f598299362032340ed: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:41.596143 8427 tablet_bootstrap.cc:492] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 41ef31ce05104786bad477cb8fd4bbd5: No bootstrap required, opened a new log
I20260929 01:02:41.596243 8427 ts_tablet_manager.cc:1397] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 41ef31ce05104786bad477cb8fd4bbd5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 01:02:41.596781 8427 raft_consensus.cc:348] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:41.596896 8425 tablet_bootstrap.cc:492] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 609dfbb102b049f598299362032340ed: No bootstrap required, opened a new log
I20260929 01:02:41.596909 8427 raft_consensus.cc:374] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:41.596946 8427 raft_consensus.cc:729] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41ef31ce05104786bad477cb8fd4bbd5, State: Initialized, Role: FOLLOWER
I20260929 01:02:41.596972 8425 ts_tablet_manager.cc:1397] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 609dfbb102b049f598299362032340ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 01:02:41.597075 8427 consensus_queue.cc:260] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 41ef31ce05104786bad477cb8fd4bbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:41.597389 8427 ts_tablet_manager.cc:1428] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 41ef31ce05104786bad477cb8fd4bbd5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 01:02:41.597349 8431 tablet_replica.cc:439] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 41ef31ce05104786bad477cb8fd4bbd5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } } }
I20260929 01:02:41.597479 8431 tablet_replica.cc:442] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 41ef31ce05104786bad477cb8fd4bbd5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:02:41.597463 8425 raft_consensus.cc:348] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 609dfbb102b049f598299362032340ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:41.597565 8425 raft_consensus.cc:374] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 609dfbb102b049f598299362032340ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:41.597590 8425 raft_consensus.cc:729] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 609dfbb102b049f598299362032340ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 609dfbb102b049f598299362032340ed, State: Initialized, Role: FOLLOWER
I20260929 01:02:41.597723 8425 consensus_queue.cc:260] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 609dfbb102b049f598299362032340ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:41.597978 8433 tablet_replica.cc:439] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 609dfbb102b049f598299362032340ed: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } } }
I20260929 01:02:41.598119 8433 tablet_replica.cc:442] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 609dfbb102b049f598299362032340ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:02:41.597980 8425 ts_tablet_manager.cc:1428] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 609dfbb102b049f598299362032340ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 01:02:41.598377 8426 tablet_bootstrap.cc:492] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e: No bootstrap required, opened a new log
I20260929 01:02:41.598459 8426 ts_tablet_manager.cc:1397] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260929 01:02:41.598869 8426 raft_consensus.cc:348] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:41.598989 8426 raft_consensus.cc:374] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:41.599026 8426 raft_consensus.cc:729] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b0eb54a51d43078a7747d7edf55c8e, State: Initialized, Role: FOLLOWER
I20260929 01:02:41.599159 8426 consensus_queue.cc:260] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:41.599393 8436 tablet_replica.cc:439] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } } }
I20260929 01:02:41.599452 8426 ts_tablet_manager.cc:1428] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:41.599525 8436 tablet_replica.cc:442] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:02:41.764222 8436 raft_consensus.cc:482] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:02:41.764355 8436 raft_consensus.cc:504] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:41.764778 8436 leader_election.cc:288] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 609dfbb102b049f598299362032340ed (127.0.98.2:42457), 41ef31ce05104786bad477cb8fd4bbd5 (127.0.98.3:33501)
I20260929 01:02:41.765233 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45bb7d5583874a43bb6aa4af4c0bf7c9" candidate_uuid: "48b0eb54a51d43078a7747d7edf55c8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "609dfbb102b049f598299362032340ed" is_pre_election: true
I20260929 01:02:41.765408 8211 raft_consensus.cc:2454] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 609dfbb102b049f598299362032340ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48b0eb54a51d43078a7747d7edf55c8e in term 0.
I20260929 01:02:41.765444 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45bb7d5583874a43bb6aa4af4c0bf7c9" candidate_uuid: "48b0eb54a51d43078a7747d7edf55c8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41ef31ce05104786bad477cb8fd4bbd5" is_pre_election: true
I20260929 01:02:41.765625 8347 raft_consensus.cc:2454] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48b0eb54a51d43078a7747d7edf55c8e in term 0.
I20260929 01:02:41.765751 8004 leader_election.cc:302] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48b0eb54a51d43078a7747d7edf55c8e, 609dfbb102b049f598299362032340ed; no voters:
I20260929 01:02:41.765995 8436 raft_consensus.cc:2781] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:02:41.766069 8436 raft_consensus.cc:482] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:02:41.766134 8436 raft_consensus.cc:3037] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:41.767139 8436 raft_consensus.cc:504] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:41.767478 8436 leader_election.cc:288] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e [CANDIDATE]: Term 1 election: Requested vote from peers 609dfbb102b049f598299362032340ed (127.0.98.2:42457), 41ef31ce05104786bad477cb8fd4bbd5 (127.0.98.3:33501)
I20260929 01:02:41.767834 8347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45bb7d5583874a43bb6aa4af4c0bf7c9" candidate_uuid: "48b0eb54a51d43078a7747d7edf55c8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41ef31ce05104786bad477cb8fd4bbd5"
I20260929 01:02:41.767834 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45bb7d5583874a43bb6aa4af4c0bf7c9" candidate_uuid: "48b0eb54a51d43078a7747d7edf55c8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "609dfbb102b049f598299362032340ed"
I20260929 01:02:41.767959 8211 raft_consensus.cc:3037] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 609dfbb102b049f598299362032340ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:41.767959 8347 raft_consensus.cc:3037] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:41.769024 8211 raft_consensus.cc:2454] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 609dfbb102b049f598299362032340ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48b0eb54a51d43078a7747d7edf55c8e in term 1.
I20260929 01:02:41.769024 8347 raft_consensus.cc:2454] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 41ef31ce05104786bad477cb8fd4bbd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48b0eb54a51d43078a7747d7edf55c8e in term 1.
I20260929 01:02:41.769326 8004 leader_election.cc:302] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48b0eb54a51d43078a7747d7edf55c8e, 609dfbb102b049f598299362032340ed; no voters:
I20260929 01:02:41.769507 8436 raft_consensus.cc:2781] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:41.769620 8436 raft_consensus.cc:686] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e [term 1 LEADER]: Becoming Leader. State: Replica: 48b0eb54a51d43078a7747d7edf55c8e, State: Running, Role: LEADER
I20260929 01:02:41.769754 8436 consensus_queue.cc:237] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:41.770351 8432 tablet_replica.cc:439] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e: TxnStatusTablet state changed. Reason: New leader 48b0eb54a51d43078a7747d7edf55c8e. Latest consensus state: current_term: 1 leader_uuid: "48b0eb54a51d43078a7747d7edf55c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } } }
I20260929 01:02:41.770466 8432 tablet_replica.cc:442] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:02:41.770750 8466 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 01:02:41.770977 8466 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 01:02:41.771395 7914 catalog_manager.cc:5495] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 48b0eb54a51d43078a7747d7edf55c8e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "48b0eb54a51d43078a7747d7edf55c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } health_report { overall_health: UNKNOWN } } }
I20260929 01:02:41.788420 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:41.784807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38557 (local address 127.0.0.1:56826)
0929 01:02:41.784980 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:41.784993 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:41.785076 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:41.785391 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:41.785393 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:41.785402 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:02:41.785493 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:41.785497 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.786481 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.786483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:41.787181 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:41.787191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.787715 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.787717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:41.787776 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:41.787815 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:02:41.788128 (+ 313us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:02:41.788142 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 01:02:41.788201 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 01:02:41.788305 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 01:02:41.788421 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:41.784892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56826 (local address 127.0.98.1:38557)
0929 01:02:41.785029 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:41.785042 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:41.785057 (+ 15us) server_negotiation.cc:407] Connection header received
0929 01:02:41.785191 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:41.785194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:41.785246 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:41.785327 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:02:41.785602 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.786375 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.787281 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.787610 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.787716 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:41.787934 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:02:41.788042 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:02:41.788109 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:41.788237 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 01:02:41.788290 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:02:41.790987 8211 raft_consensus.cc:1261] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 609dfbb102b049f598299362032340ed [term 1 FOLLOWER]: Refusing update from remote peer 48b0eb54a51d43078a7747d7edf55c8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:41.791018 8347 raft_consensus.cc:1261] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 41ef31ce05104786bad477cb8fd4bbd5 [term 1 FOLLOWER]: Refusing update from remote peer 48b0eb54a51d43078a7747d7edf55c8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:41.791421 8432 consensus_queue.cc:1035] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:41.791555 8436 consensus_queue.cc:1035] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:41.792937 8433 tablet_replica.cc:439] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 609dfbb102b049f598299362032340ed: TxnStatusTablet state changed. Reason: New leader 48b0eb54a51d43078a7747d7edf55c8e. Latest consensus state: current_term: 1 leader_uuid: "48b0eb54a51d43078a7747d7edf55c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } } }
I20260929 01:02:41.793056 8433 tablet_replica.cc:442] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 609dfbb102b049f598299362032340ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:02:41.793082 8431 tablet_replica.cc:439] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 41ef31ce05104786bad477cb8fd4bbd5: TxnStatusTablet state changed. Reason: New leader 48b0eb54a51d43078a7747d7edf55c8e. Latest consensus state: current_term: 1 leader_uuid: "48b0eb54a51d43078a7747d7edf55c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } } }
I20260929 01:02:41.793184 8431 tablet_replica.cc:442] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 41ef31ce05104786bad477cb8fd4bbd5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:02:41.795101 8436 tablet_replica.cc:439] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "48b0eb54a51d43078a7747d7edf55c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } } }
I20260929 01:02:41.795209 8436 tablet_replica.cc:442] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:02:41.796583 8431 tablet_replica.cc:439] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 41ef31ce05104786bad477cb8fd4bbd5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "48b0eb54a51d43078a7747d7edf55c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } } }
I20260929 01:02:41.796684 8431 tablet_replica.cc:442] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 41ef31ce05104786bad477cb8fd4bbd5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:02:41.797745 8436 tablet_replica.cc:439] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "48b0eb54a51d43078a7747d7edf55c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } } }
I20260929 01:02:41.797851 8436 tablet_replica.cc:442] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 48b0eb54a51d43078a7747d7edf55c8e: This TxnStatusTablet replica's current role is: LEADER
I20260929 01:02:41.799005 8433 tablet_replica.cc:439] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 609dfbb102b049f598299362032340ed: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "48b0eb54a51d43078a7747d7edf55c8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } } }
I20260929 01:02:41.799124 8433 tablet_replica.cc:442] T 45bb7d5583874a43bb6aa4af4c0bf7c9 P 609dfbb102b049f598299362032340ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 01:02:41.799199 8441 mvcc.cc:205] Tried to move back new op lower bound from 7334476848291622912 to 7334476848210046976. Current Snapshot: MvccSnapshot[applied={T|T < 7334476848291622912 or (T in {7334476848291622912})}]
I20260929 01:02:41.809356 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:41.805710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41055 (local address 127.0.98.1:38557)
0929 01:02:41.805841 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:41.805845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:41.806052 (+ 207us) server_negotiation.cc:407] Connection header received
0929 01:02:41.806170 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:41.806172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:41.806210 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:41.806268 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:02:41.806541 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.807140 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.808351 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.808592 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.808675 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:41.808958 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:02:41.809025 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:02:41.809078 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:41.809181 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 01:02:41.809226 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":54}
I20260929 01:02:41.809461 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:41.805657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38557 (local address 127.0.98.1:41055)
0929 01:02:41.805962 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:41.805976 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:41.806082 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:41.806297 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:41.806300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:41.806307 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:02:41.806443 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:41.806450 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.807235 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.807238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:41.808247 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:41.808258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.808682 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.808686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:41.808788 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:41.808836 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:02:41.809103 (+ 267us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:02:41.809109 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 01:02:41.809178 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 01:02:41.809265 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":19,"thread_start_us":114,"threads_started":1}
I20260929 01:02:41.816032 8211 raft_consensus.cc:1261] T 580b6f5520c34d9d8a2e04cd7998e349 P 609dfbb102b049f598299362032340ed [term 1 FOLLOWER]: Refusing update from remote peer 48b0eb54a51d43078a7747d7edf55c8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:41.816073 8347 raft_consensus.cc:1261] T 580b6f5520c34d9d8a2e04cd7998e349 P 41ef31ce05104786bad477cb8fd4bbd5 [term 1 FOLLOWER]: Refusing update from remote peer 48b0eb54a51d43078a7747d7edf55c8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:41.816422 8432 consensus_queue.cc:1035] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:41.816622 8436 consensus_queue.cc:1035] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:41.820822 8439 mvcc.cc:205] Tried to move back new op lower bound from 7334476848394534912 to 7334476847226937344. Current Snapshot: MvccSnapshot[applied={T|T < 7334476848394534912}]
I20260929 01:02:41.856181 7914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:41.856353 7914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:41.858214 7914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c: Sending DeleteTablet for 3 replicas of tablet 580b6f5520c34d9d8a2e04cd7998e349
I20260929 01:02:41.858865 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet 580b6f5520c34d9d8a2e04cd7998e349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:41 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48188
I20260929 01:02:41.858939 8330 tablet_service.cc:1514] Processing DeleteTablet for tablet 580b6f5520c34d9d8a2e04cd7998e349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:41 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57930
I20260929 01:02:41.859127 8450 tablet_replica.cc:330] T 580b6f5520c34d9d8a2e04cd7998e349 P 609dfbb102b049f598299362032340ed: stopping tablet replica
I20260929 01:02:41.859223 8449 tablet_replica.cc:330] T 580b6f5520c34d9d8a2e04cd7998e349 P 41ef31ce05104786bad477cb8fd4bbd5: stopping tablet replica
I20260929 01:02:41.859261 8450 raft_consensus.cc:2229] T 580b6f5520c34d9d8a2e04cd7998e349 P 609dfbb102b049f598299362032340ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:41.859318 8449 raft_consensus.cc:2229] T 580b6f5520c34d9d8a2e04cd7998e349 P 41ef31ce05104786bad477cb8fd4bbd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:41.859365 8450 raft_consensus.cc:2258] T 580b6f5520c34d9d8a2e04cd7998e349 P 609dfbb102b049f598299362032340ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:41.859426 8449 raft_consensus.cc:2258] T 580b6f5520c34d9d8a2e04cd7998e349 P 41ef31ce05104786bad477cb8fd4bbd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:41.860081 8450 ts_tablet_manager.cc:1905] T 580b6f5520c34d9d8a2e04cd7998e349 P 609dfbb102b049f598299362032340ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:41.860375 8449 ts_tablet_manager.cc:1905] T 580b6f5520c34d9d8a2e04cd7998e349 P 41ef31ce05104786bad477cb8fd4bbd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:41.862440 8449 ts_tablet_manager.cc:1918] T 580b6f5520c34d9d8a2e04cd7998e349 P 41ef31ce05104786bad477cb8fd4bbd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:02:41.862531 8449 log.cc:1196] T 580b6f5520c34d9d8a2e04cd7998e349 P 41ef31ce05104786bad477cb8fd4bbd5: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/580b6f5520c34d9d8a2e04cd7998e349
I20260929 01:02:41.862783 8449 ts_tablet_manager.cc:1939] T 580b6f5520c34d9d8a2e04cd7998e349 P 41ef31ce05104786bad477cb8fd4bbd5: Deleting consensus metadata
I20260929 01:02:41.863044 8450 ts_tablet_manager.cc:1918] T 580b6f5520c34d9d8a2e04cd7998e349 P 609dfbb102b049f598299362032340ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:02:41.863122 8450 log.cc:1196] T 580b6f5520c34d9d8a2e04cd7998e349 P 609dfbb102b049f598299362032340ed: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-1/wal/wals/580b6f5520c34d9d8a2e04cd7998e349
I20260929 01:02:41.863408 8450 ts_tablet_manager.cc:1939] T 580b6f5520c34d9d8a2e04cd7998e349 P 609dfbb102b049f598299362032340ed: Deleting consensus metadata
I20260929 01:02:41.863509 7899 catalog_manager.cc:4841] TS 41ef31ce05104786bad477cb8fd4bbd5 (127.0.98.3:33501): tablet 580b6f5520c34d9d8a2e04cd7998e349 (table test-table [id=2afee3385a2c4226b8510d43399f1447]) successfully deleted
I20260929 01:02:41.863583 8050 tablet_service.cc:1514] Processing DeleteTablet for tablet 580b6f5520c34d9d8a2e04cd7998e349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:41 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56970
I20260929 01:02:41.863775 8448 tablet_replica.cc:330] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e: stopping tablet replica
I20260929 01:02:41.863915 8448 raft_consensus.cc:2229] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:02:41.864043 7900 catalog_manager.cc:4841] TS 609dfbb102b049f598299362032340ed (127.0.98.2:42457): tablet 580b6f5520c34d9d8a2e04cd7998e349 (table test-table [id=2afee3385a2c4226b8510d43399f1447]) successfully deleted
I20260929 01:02:41.864151 8448 raft_consensus.cc:2258] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:41.864953 8448 ts_tablet_manager.cc:1905] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:41.867728 8448 ts_tablet_manager.cc:1918] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:02:41.867779 8448 log.cc:1196] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-0/wal/wals/580b6f5520c34d9d8a2e04cd7998e349
I20260929 01:02:41.867969 8448 ts_tablet_manager.cc:1939] T 580b6f5520c34d9d8a2e04cd7998e349 P 48b0eb54a51d43078a7747d7edf55c8e: Deleting consensus metadata
I20260929 01:02:41.868718 7899 catalog_manager.cc:4841] TS 48b0eb54a51d43078a7747d7edf55c8e (127.0.98.1:38557): tablet 580b6f5520c34d9d8a2e04cd7998e349 (table test-table [id=2afee3385a2c4226b8510d43399f1447]) successfully deleted
Sep 29 01:02:41 dist-test-slave-7zdl krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 01:02:41 dist-test-slave-7zdl krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:41 dist-test-slave-7zdl krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 01:02:41.894107 8461 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:41.887441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54430 (local address 127.0.98.62:42293)
0929 01:02:41.887597 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:41.887603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:41.887773 (+ 170us) server_negotiation.cc:407] Connection header received
0929 01:02:41.887954 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:41.887957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:41.888000 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:41.888086 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:41.888788 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.889402 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.890393 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.890620 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.890730 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:41.893984 (+ 3254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54430: BlockingRecv error: recv got EOF from 127.0.0.1:54430 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 01:02:41 dist-test-slave-7zdl krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 01:02:41.903823 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:41.895703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54440 (local address 127.0.98.62:42293)
0929 01:02:41.895812 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:41.895816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:41.896021 (+ 205us) server_negotiation.cc:407] Connection header received
0929 01:02:41.896164 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:41.896167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:41.896213 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:41.896308 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:41.896881 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.897388 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.898181 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:41.898411 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:41.898514 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:41.900758 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:41.900762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:41.900765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:41.900789 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:41.902198 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:41.902618 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:41.902622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:41.902623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:41.902691 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:41.903013 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:41.903018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:41.903020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:41.903189 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:41.903321 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:41.903546 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 01:02:41.903665 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 01:02:41.906894 7914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:02:41.907431 7914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:02:41.913902 8330 tablet_service.cc:1467] Processing CreateTablet for tablet 3882824902e548afba33edf247c44ed1 (DEFAULT_TABLE table=test-table [id=56c05ca78d2c42a6b34a811b48b41c22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:41.913950 8050 tablet_service.cc:1467] Processing CreateTablet for tablet 3882824902e548afba33edf247c44ed1 (DEFAULT_TABLE table=test-table [id=56c05ca78d2c42a6b34a811b48b41c22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:41.914242 8330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3882824902e548afba33edf247c44ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:41.914256 8050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3882824902e548afba33edf247c44ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:41.915014 8191 tablet_service.cc:1467] Processing CreateTablet for tablet 3882824902e548afba33edf247c44ed1 (DEFAULT_TABLE table=test-table [id=56c05ca78d2c42a6b34a811b48b41c22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:41.915302 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3882824902e548afba33edf247c44ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:41.917032 8426 tablet_bootstrap.cc:492] T 3882824902e548afba33edf247c44ed1 P 48b0eb54a51d43078a7747d7edf55c8e: Bootstrap starting.
I20260929 01:02:41.917780 8425 tablet_bootstrap.cc:492] T 3882824902e548afba33edf247c44ed1 P 609dfbb102b049f598299362032340ed: Bootstrap starting.
I20260929 01:02:41.918232 8426 tablet_bootstrap.cc:654] T 3882824902e548afba33edf247c44ed1 P 48b0eb54a51d43078a7747d7edf55c8e: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:41.918690 8425 tablet_bootstrap.cc:654] T 3882824902e548afba33edf247c44ed1 P 609dfbb102b049f598299362032340ed: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:41.919503 8426 tablet_bootstrap.cc:492] T 3882824902e548afba33edf247c44ed1 P 48b0eb54a51d43078a7747d7edf55c8e: No bootstrap required, opened a new log
I20260929 01:02:41.919606 8426 ts_tablet_manager.cc:1397] T 3882824902e548afba33edf247c44ed1 P 48b0eb54a51d43078a7747d7edf55c8e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 01:02:41.920090 8426 raft_consensus.cc:348] T 3882824902e548afba33edf247c44ed1 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:41.920207 8426 raft_consensus.cc:374] T 3882824902e548afba33edf247c44ed1 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:41.920245 8426 raft_consensus.cc:729] T 3882824902e548afba33edf247c44ed1 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b0eb54a51d43078a7747d7edf55c8e, State: Initialized, Role: FOLLOWER
I20260929 01:02:41.920358 8426 consensus_queue.cc:260] T 3882824902e548afba33edf247c44ed1 P 48b0eb54a51d43078a7747d7edf55c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:41.920627 8426 ts_tablet_manager.cc:1428] T 3882824902e548afba33edf247c44ed1 P 48b0eb54a51d43078a7747d7edf55c8e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:41.922284 8425 tablet_bootstrap.cc:492] T 3882824902e548afba33edf247c44ed1 P 609dfbb102b049f598299362032340ed: No bootstrap required, opened a new log
I20260929 01:02:41.922385 8425 ts_tablet_manager.cc:1397] T 3882824902e548afba33edf247c44ed1 P 609dfbb102b049f598299362032340ed: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260929 01:02:41.922839 8425 raft_consensus.cc:348] T 3882824902e548afba33edf247c44ed1 P 609dfbb102b049f598299362032340ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:41.922945 8425 raft_consensus.cc:374] T 3882824902e548afba33edf247c44ed1 P 609dfbb102b049f598299362032340ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:41.922972 8425 raft_consensus.cc:729] T 3882824902e548afba33edf247c44ed1 P 609dfbb102b049f598299362032340ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 609dfbb102b049f598299362032340ed, State: Initialized, Role: FOLLOWER
I20260929 01:02:41.923084 8425 consensus_queue.cc:260] T 3882824902e548afba33edf247c44ed1 P 609dfbb102b049f598299362032340ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:41.923331 8425 ts_tablet_manager.cc:1428] T 3882824902e548afba33edf247c44ed1 P 609dfbb102b049f598299362032340ed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:41.923610 8427 tablet_bootstrap.cc:492] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5: Bootstrap starting.
I20260929 01:02:41.924646 8427 tablet_bootstrap.cc:654] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:41.925803 8427 tablet_bootstrap.cc:492] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5: No bootstrap required, opened a new log
I20260929 01:02:41.925894 8427 ts_tablet_manager.cc:1397] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:41.926328 8427 raft_consensus.cc:348] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:41.926445 8427 raft_consensus.cc:374] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:41.926483 8427 raft_consensus.cc:729] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41ef31ce05104786bad477cb8fd4bbd5, State: Initialized, Role: FOLLOWER
I20260929 01:02:41.926613 8427 consensus_queue.cc:260] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:41.926928 8427 ts_tablet_manager.cc:1428] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:41.999696 8431 raft_consensus.cc:482] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:02:41.999831 8431 raft_consensus.cc:504] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:42.000996 8431 leader_election.cc:288] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48b0eb54a51d43078a7747d7edf55c8e (127.0.98.1:38557), 609dfbb102b049f598299362032340ed (127.0.98.2:42457)
I20260929 01:02:42.004808 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:42.001330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48529 (local address 127.0.98.1:38557)
0929 01:02:42.001478 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:42.001484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:42.001605 (+ 121us) server_negotiation.cc:407] Connection header received
0929 01:02:42.001824 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:42.001827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:42.001883 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:42.001969 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:42.002233 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:42.002735 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:42.003845 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:42.004258 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:42.004387 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:42.004451 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:42.004626 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 01:02:42.004693 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 01:02:42.004827 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:42.001204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38557 (local address 127.0.98.3:48529)
0929 01:02:42.001523 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:42.001538 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:42.001686 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:42.001984 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:42.001987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:42.002005 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:42.002118 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:42.002123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:42.002909 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:42.002912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:42.003721 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:42.003729 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:42.004389 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:42.004393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:42.004499 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:42.004519 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 01:02:42.004609 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 01:02:42.004673 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
I20260929 01:02:42.004962 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:42.001561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43361 (local address 127.0.98.2:42457)
0929 01:02:42.001817 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:42.001823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:42.001838 (+ 15us) server_negotiation.cc:407] Connection header received
0929 01:02:42.001891 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:42.001894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:42.001959 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:42.002045 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:02:42.002333 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:42.002923 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:42.003957 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:42.004349 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:42.004469 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:42.004531 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:42.004708 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 01:02:42.004774 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":72,"threads_started":1}
I20260929 01:02:42.005152 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:42.001215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42457 (local address 127.0.98.3:43361)
0929 01:02:42.001618 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:42.001631 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:42.001692 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:42.002100 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:42.002103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:42.002124 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:02:42.002213 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:42.002217 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:42.003029 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:42.003035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:42.003804 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:42.003812 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:42.004471 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:42.004474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:42.004580 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:42.004606 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 01:02:42.004693 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 01:02:42.004757 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":17,"thread_start_us":74,"threads_started":1}
I20260929 01:02:42.005300 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3882824902e548afba33edf247c44ed1" candidate_uuid: "41ef31ce05104786bad477cb8fd4bbd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48b0eb54a51d43078a7747d7edf55c8e" is_pre_election: true
I20260929 01:02:42.005565 8071 raft_consensus.cc:2454] T 3882824902e548afba33edf247c44ed1 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41ef31ce05104786bad477cb8fd4bbd5 in term 0.
I20260929 01:02:42.005563 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3882824902e548afba33edf247c44ed1" candidate_uuid: "41ef31ce05104786bad477cb8fd4bbd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "609dfbb102b049f598299362032340ed" is_pre_election: true
I20260929 01:02:42.005702 8211 raft_consensus.cc:2454] T 3882824902e548afba33edf247c44ed1 P 609dfbb102b049f598299362032340ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41ef31ce05104786bad477cb8fd4bbd5 in term 0.
I20260929 01:02:42.006006 8283 leader_election.cc:302] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 41ef31ce05104786bad477cb8fd4bbd5, 48b0eb54a51d43078a7747d7edf55c8e, 609dfbb102b049f598299362032340ed; no voters:
I20260929 01:02:42.006266 8431 raft_consensus.cc:2781] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:02:42.006335 8431 raft_consensus.cc:482] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:02:42.006377 8431 raft_consensus.cc:3037] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:42.007479 8431 raft_consensus.cc:504] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:42.007942 8431 leader_election.cc:288] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5 [CANDIDATE]: Term 1 election: Requested vote from peers 48b0eb54a51d43078a7747d7edf55c8e (127.0.98.1:38557), 609dfbb102b049f598299362032340ed (127.0.98.2:42457)
I20260929 01:02:42.008394 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3882824902e548afba33edf247c44ed1" candidate_uuid: "41ef31ce05104786bad477cb8fd4bbd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "609dfbb102b049f598299362032340ed"
I20260929 01:02:42.008499 8211 raft_consensus.cc:3037] T 3882824902e548afba33edf247c44ed1 P 609dfbb102b049f598299362032340ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:42.009331 8211 raft_consensus.cc:2454] T 3882824902e548afba33edf247c44ed1 P 609dfbb102b049f598299362032340ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41ef31ce05104786bad477cb8fd4bbd5 in term 1.
I20260929 01:02:42.009613 8284 leader_election.cc:302] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41ef31ce05104786bad477cb8fd4bbd5, 609dfbb102b049f598299362032340ed; no voters:
I20260929 01:02:42.009578 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3882824902e548afba33edf247c44ed1" candidate_uuid: "41ef31ce05104786bad477cb8fd4bbd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48b0eb54a51d43078a7747d7edf55c8e"
I20260929 01:02:42.009729 8071 raft_consensus.cc:3037] T 3882824902e548afba33edf247c44ed1 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:42.009771 8431 raft_consensus.cc:2781] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:42.009969 8431 raft_consensus.cc:686] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5 [term 1 LEADER]: Becoming Leader. State: Replica: 41ef31ce05104786bad477cb8fd4bbd5, State: Running, Role: LEADER
I20260929 01:02:42.010208 8431 consensus_queue.cc:237] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:42.010848 8071 raft_consensus.cc:2454] T 3882824902e548afba33edf247c44ed1 P 48b0eb54a51d43078a7747d7edf55c8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41ef31ce05104786bad477cb8fd4bbd5 in term 1.
I20260929 01:02:42.012465 7914 catalog_manager.cc:5495] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41ef31ce05104786bad477cb8fd4bbd5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "41ef31ce05104786bad477cb8fd4bbd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } health_report { overall_health: HEALTHY } } }
I20260929 01:02:42.028008 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:42.024152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50192 (local address 127.0.98.3:33501)
0929 01:02:42.024410 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:42.024416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:42.024427 (+ 11us) server_negotiation.cc:407] Connection header received
0929 01:02:42.024485 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:42.024487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:42.024533 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:42.024629 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:02:42.024911 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:42.025852 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:42.026653 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:42.026955 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:42.027033 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:42.027250 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:02:42.027685 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:02:42.027768 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:42.027848 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 01:02:42.027893 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":117,"threads_started":1}
I20260929 01:02:42.030701 8211 raft_consensus.cc:1261] T 3882824902e548afba33edf247c44ed1 P 609dfbb102b049f598299362032340ed [term 1 FOLLOWER]: Refusing update from remote peer 41ef31ce05104786bad477cb8fd4bbd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:42.030889 8071 raft_consensus.cc:1261] T 3882824902e548afba33edf247c44ed1 P 48b0eb54a51d43078a7747d7edf55c8e [term 1 FOLLOWER]: Refusing update from remote peer 41ef31ce05104786bad477cb8fd4bbd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:42.031221 8431 consensus_queue.cc:1035] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:42.031386 8496 consensus_queue.cc:1035] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:42.042030 7914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:42.042174 7914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:42.044227 7914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c: Sending DeleteTablet for 3 replicas of tablet 3882824902e548afba33edf247c44ed1
I20260929 01:02:42.044796 8050 tablet_service.cc:1514] Processing DeleteTablet for tablet 3882824902e548afba33edf247c44ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:42 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56970
I20260929 01:02:42.044883 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet 3882824902e548afba33edf247c44ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:42 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48188
I20260929 01:02:42.045101 8330 tablet_service.cc:1514] Processing DeleteTablet for tablet 3882824902e548afba33edf247c44ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:42 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57930
I20260929 01:02:42.045295 8448 tablet_replica.cc:330] T 3882824902e548afba33edf247c44ed1 P 48b0eb54a51d43078a7747d7edf55c8e: stopping tablet replica
I20260929 01:02:42.045428 8448 raft_consensus.cc:2229] T 3882824902e548afba33edf247c44ed1 P 48b0eb54a51d43078a7747d7edf55c8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:42.045564 8448 raft_consensus.cc:2258] T 3882824902e548afba33edf247c44ed1 P 48b0eb54a51d43078a7747d7edf55c8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:42.045745 8450 tablet_replica.cc:330] T 3882824902e548afba33edf247c44ed1 P 609dfbb102b049f598299362032340ed: stopping tablet replica
I20260929 01:02:42.045899 8450 raft_consensus.cc:2229] T 3882824902e548afba33edf247c44ed1 P 609dfbb102b049f598299362032340ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:42.046060 8450 raft_consensus.cc:2258] T 3882824902e548afba33edf247c44ed1 P 609dfbb102b049f598299362032340ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:42.046212 8449 tablet_replica.cc:330] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5: stopping tablet replica
I20260929 01:02:42.046308 8449 raft_consensus.cc:2229] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:02:42.046550 8449 raft_consensus.cc:2258] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:42.047375 7914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 01:02:42.047566 8449 ts_tablet_manager.cc:1905] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:42.047575 8448 ts_tablet_manager.cc:1905] T 3882824902e548afba33edf247c44ed1 P 48b0eb54a51d43078a7747d7edf55c8e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 01:02:42.047760 7914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:02:42.050025 8448 ts_tablet_manager.cc:1918] T 3882824902e548afba33edf247c44ed1 P 48b0eb54a51d43078a7747d7edf55c8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:02:42.050091 8448 log.cc:1196] T 3882824902e548afba33edf247c44ed1 P 48b0eb54a51d43078a7747d7edf55c8e: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-0/wal/wals/3882824902e548afba33edf247c44ed1
I20260929 01:02:42.050086 8449 ts_tablet_manager.cc:1918] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:02:42.050158 8449 log.cc:1196] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/3882824902e548afba33edf247c44ed1
I20260929 01:02:42.050356 8448 ts_tablet_manager.cc:1939] T 3882824902e548afba33edf247c44ed1 P 48b0eb54a51d43078a7747d7edf55c8e: Deleting consensus metadata
I20260929 01:02:42.050405 8449 ts_tablet_manager.cc:1939] T 3882824902e548afba33edf247c44ed1 P 41ef31ce05104786bad477cb8fd4bbd5: Deleting consensus metadata
I20260929 01:02:42.051008 7899 catalog_manager.cc:4841] TS 48b0eb54a51d43078a7747d7edf55c8e (127.0.98.1:38557): tablet 3882824902e548afba33edf247c44ed1 (table test-table [id=56c05ca78d2c42a6b34a811b48b41c22]) successfully deleted
I20260929 01:02:42.051479 8450 ts_tablet_manager.cc:1905] T 3882824902e548afba33edf247c44ed1 P 609dfbb102b049f598299362032340ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:42.051752 7899 catalog_manager.cc:4841] TS 41ef31ce05104786bad477cb8fd4bbd5 (127.0.98.3:33501): tablet 3882824902e548afba33edf247c44ed1 (table test-table [id=56c05ca78d2c42a6b34a811b48b41c22]) successfully deleted
I20260929 01:02:42.053715 8191 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf449bdd2d84317beaba14cbcbc6a31 (DEFAULT_TABLE table=test-table [id=e905fa812f854923b62f1c00db1c50fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:42.053779 8330 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf449bdd2d84317beaba14cbcbc6a31 (DEFAULT_TABLE table=test-table [id=e905fa812f854923b62f1c00db1c50fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:42.053975 8330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bf449bdd2d84317beaba14cbcbc6a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:42.053975 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bf449bdd2d84317beaba14cbcbc6a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:42.054427 8050 tablet_service.cc:1467] Processing CreateTablet for tablet 2bf449bdd2d84317beaba14cbcbc6a31 (DEFAULT_TABLE table=test-table [id=e905fa812f854923b62f1c00db1c50fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:02:42.054641 8050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bf449bdd2d84317beaba14cbcbc6a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:42.056015 8427 tablet_bootstrap.cc:492] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5: Bootstrap starting.
I20260929 01:02:42.056034 8425 tablet_bootstrap.cc:492] T 2bf449bdd2d84317beaba14cbcbc6a31 P 609dfbb102b049f598299362032340ed: Bootstrap starting.
I20260929 01:02:42.056717 8426 tablet_bootstrap.cc:492] T 2bf449bdd2d84317beaba14cbcbc6a31 P 48b0eb54a51d43078a7747d7edf55c8e: Bootstrap starting.
I20260929 01:02:42.056948 8427 tablet_bootstrap.cc:654] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:42.057046 8425 tablet_bootstrap.cc:654] T 2bf449bdd2d84317beaba14cbcbc6a31 P 609dfbb102b049f598299362032340ed: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:42.057399 8426 tablet_bootstrap.cc:654] T 2bf449bdd2d84317beaba14cbcbc6a31 P 48b0eb54a51d43078a7747d7edf55c8e: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:42.053723 8450 ts_tablet_manager.cc:1918] T 3882824902e548afba33edf247c44ed1 P 609dfbb102b049f598299362032340ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:02:42.058121 8450 log.cc:1196] T 3882824902e548afba33edf247c44ed1 P 609dfbb102b049f598299362032340ed: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-1/wal/wals/3882824902e548afba33edf247c44ed1
I20260929 01:02:42.058285 8427 tablet_bootstrap.cc:492] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5: No bootstrap required, opened a new log
I20260929 01:02:42.058349 8425 tablet_bootstrap.cc:492] T 2bf449bdd2d84317beaba14cbcbc6a31 P 609dfbb102b049f598299362032340ed: No bootstrap required, opened a new log
I20260929 01:02:42.058353 8427 ts_tablet_manager.cc:1397] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:42.058380 8450 ts_tablet_manager.cc:1939] T 3882824902e548afba33edf247c44ed1 P 609dfbb102b049f598299362032340ed: Deleting consensus metadata
I20260929 01:02:42.058410 8425 ts_tablet_manager.cc:1397] T 2bf449bdd2d84317beaba14cbcbc6a31 P 609dfbb102b049f598299362032340ed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:42.058763 8427 raft_consensus.cc:348] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:42.058771 8425 raft_consensus.cc:348] T 2bf449bdd2d84317beaba14cbcbc6a31 P 609dfbb102b049f598299362032340ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:42.058889 8427 raft_consensus.cc:374] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:42.058889 8425 raft_consensus.cc:374] T 2bf449bdd2d84317beaba14cbcbc6a31 P 609dfbb102b049f598299362032340ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:42.058923 8427 raft_consensus.cc:729] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41ef31ce05104786bad477cb8fd4bbd5, State: Initialized, Role: FOLLOWER
I20260929 01:02:42.058935 8425 raft_consensus.cc:729] T 2bf449bdd2d84317beaba14cbcbc6a31 P 609dfbb102b049f598299362032340ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 609dfbb102b049f598299362032340ed, State: Initialized, Role: FOLLOWER
I20260929 01:02:42.058924 7900 catalog_manager.cc:4841] TS 609dfbb102b049f598299362032340ed (127.0.98.2:42457): tablet 3882824902e548afba33edf247c44ed1 (table test-table [id=56c05ca78d2c42a6b34a811b48b41c22]) successfully deleted
I20260929 01:02:42.059039 8427 consensus_queue.cc:260] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:42.059056 8425 consensus_queue.cc:260] T 2bf449bdd2d84317beaba14cbcbc6a31 P 609dfbb102b049f598299362032340ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:42.059291 8427 ts_tablet_manager.cc:1428] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:42.059291 8425 ts_tablet_manager.cc:1428] T 2bf449bdd2d84317beaba14cbcbc6a31 P 609dfbb102b049f598299362032340ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 01:02:42.059937 8426 tablet_bootstrap.cc:492] T 2bf449bdd2d84317beaba14cbcbc6a31 P 48b0eb54a51d43078a7747d7edf55c8e: No bootstrap required, opened a new log
I20260929 01:02:42.060007 8426 ts_tablet_manager.cc:1397] T 2bf449bdd2d84317beaba14cbcbc6a31 P 48b0eb54a51d43078a7747d7edf55c8e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 01:02:42.060328 8426 raft_consensus.cc:348] T 2bf449bdd2d84317beaba14cbcbc6a31 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:42.060426 8426 raft_consensus.cc:374] T 2bf449bdd2d84317beaba14cbcbc6a31 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:42.060454 8426 raft_consensus.cc:729] T 2bf449bdd2d84317beaba14cbcbc6a31 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b0eb54a51d43078a7747d7edf55c8e, State: Initialized, Role: FOLLOWER
I20260929 01:02:42.060576 8426 consensus_queue.cc:260] T 2bf449bdd2d84317beaba14cbcbc6a31 P 48b0eb54a51d43078a7747d7edf55c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:42.060793 8426 ts_tablet_manager.cc:1428] T 2bf449bdd2d84317beaba14cbcbc6a31 P 48b0eb54a51d43078a7747d7edf55c8e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 01:02:42.127034 8502 raft_consensus.cc:482] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:02:42.127173 8502 raft_consensus.cc:504] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:42.127667 8502 leader_election.cc:288] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48b0eb54a51d43078a7747d7edf55c8e (127.0.98.1:38557), 609dfbb102b049f598299362032340ed (127.0.98.2:42457)
I20260929 01:02:42.128090 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf449bdd2d84317beaba14cbcbc6a31" candidate_uuid: "41ef31ce05104786bad477cb8fd4bbd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48b0eb54a51d43078a7747d7edf55c8e" is_pre_election: true
I20260929 01:02:42.128134 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf449bdd2d84317beaba14cbcbc6a31" candidate_uuid: "41ef31ce05104786bad477cb8fd4bbd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "609dfbb102b049f598299362032340ed" is_pre_election: true
I20260929 01:02:42.128242 8071 raft_consensus.cc:2454] T 2bf449bdd2d84317beaba14cbcbc6a31 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41ef31ce05104786bad477cb8fd4bbd5 in term 0.
I20260929 01:02:42.128293 8211 raft_consensus.cc:2454] T 2bf449bdd2d84317beaba14cbcbc6a31 P 609dfbb102b049f598299362032340ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41ef31ce05104786bad477cb8fd4bbd5 in term 0.
I20260929 01:02:42.128613 8283 leader_election.cc:302] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41ef31ce05104786bad477cb8fd4bbd5, 48b0eb54a51d43078a7747d7edf55c8e; no voters:
I20260929 01:02:42.128800 8502 raft_consensus.cc:2781] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:02:42.128883 8502 raft_consensus.cc:482] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:02:42.128937 8502 raft_consensus.cc:3037] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:42.130015 8502 raft_consensus.cc:504] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:42.130362 8502 leader_election.cc:288] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5 [CANDIDATE]: Term 1 election: Requested vote from peers 48b0eb54a51d43078a7747d7edf55c8e (127.0.98.1:38557), 609dfbb102b049f598299362032340ed (127.0.98.2:42457)
I20260929 01:02:42.130662 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf449bdd2d84317beaba14cbcbc6a31" candidate_uuid: "41ef31ce05104786bad477cb8fd4bbd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48b0eb54a51d43078a7747d7edf55c8e"
I20260929 01:02:42.130697 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bf449bdd2d84317beaba14cbcbc6a31" candidate_uuid: "41ef31ce05104786bad477cb8fd4bbd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "609dfbb102b049f598299362032340ed"
I20260929 01:02:42.130848 8211 raft_consensus.cc:3037] T 2bf449bdd2d84317beaba14cbcbc6a31 P 609dfbb102b049f598299362032340ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:42.130856 8071 raft_consensus.cc:3037] T 2bf449bdd2d84317beaba14cbcbc6a31 P 48b0eb54a51d43078a7747d7edf55c8e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:42.131748 8071 raft_consensus.cc:2454] T 2bf449bdd2d84317beaba14cbcbc6a31 P 48b0eb54a51d43078a7747d7edf55c8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41ef31ce05104786bad477cb8fd4bbd5 in term 1.
I20260929 01:02:42.131935 8211 raft_consensus.cc:2454] T 2bf449bdd2d84317beaba14cbcbc6a31 P 609dfbb102b049f598299362032340ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41ef31ce05104786bad477cb8fd4bbd5 in term 1.
I20260929 01:02:42.132026 8283 leader_election.cc:302] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41ef31ce05104786bad477cb8fd4bbd5, 48b0eb54a51d43078a7747d7edf55c8e; no voters:
I20260929 01:02:42.132192 8502 raft_consensus.cc:2781] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:42.132302 8502 raft_consensus.cc:686] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5 [term 1 LEADER]: Becoming Leader. State: Replica: 41ef31ce05104786bad477cb8fd4bbd5, State: Running, Role: LEADER
I20260929 01:02:42.132431 8502 consensus_queue.cc:237] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } }
I20260929 01:02:42.134056 7914 catalog_manager.cc:5495] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41ef31ce05104786bad477cb8fd4bbd5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "41ef31ce05104786bad477cb8fd4bbd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41ef31ce05104786bad477cb8fd4bbd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33501 } health_report { overall_health: HEALTHY } } }
I20260929 01:02:42.174691 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:42.170424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40389 (local address 127.0.98.1:38557)
0929 01:02:42.170552 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:42.170556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:42.170725 (+ 169us) server_negotiation.cc:407] Connection header received
0929 01:02:42.170902 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:42.170904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:42.170950 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:42.171012 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:02:42.171318 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:42.171946 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:42.173648 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:42.173869 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:42.173949 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:42.174288 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:02:42.174352 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:02:42.174404 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:42.174540 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 01:02:42.174584 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 01:02:42.174772 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:42.170293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38557 (local address 127.0.98.3:40389)
0929 01:02:42.170632 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:42.170648 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:42.170762 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:42.171054 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:42.171057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:42.171066 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:02:42.171206 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:42.171212 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:42.172077 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:42.172081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:42.173524 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:42.173535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:42.174015 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:42.174019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:42.174107 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:42.174158 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:02:42.174443 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:02:42.174450 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 01:02:42.174526 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 01:02:42.174604 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":135,"threads_started":1}
I20260929 01:02:42.181203 8211 raft_consensus.cc:1261] T 2bf449bdd2d84317beaba14cbcbc6a31 P 609dfbb102b049f598299362032340ed [term 1 FOLLOWER]: Refusing update from remote peer 41ef31ce05104786bad477cb8fd4bbd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:42.181339 8071 raft_consensus.cc:1261] T 2bf449bdd2d84317beaba14cbcbc6a31 P 48b0eb54a51d43078a7747d7edf55c8e [term 1 FOLLOWER]: Refusing update from remote peer 41ef31ce05104786bad477cb8fd4bbd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:02:42.181595 8431 consensus_queue.cc:1035] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "609dfbb102b049f598299362032340ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:42.181907 8431 consensus_queue.cc:1035] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48b0eb54a51d43078a7747d7edf55c8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:02:42.199188 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:42.195485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33501 (local address 127.0.98.1:52379)
0929 01:02:42.195711 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:42.195727 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:42.195801 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:42.196063 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:42.196066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:42.196074 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 01:02:42.196170 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:42.196175 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:42.197124 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:42.197127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:42.197870 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:42.197880 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:42.198466 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:42.198469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:42.198542 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:42.198569 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 01:02:42.198927 (+ 358us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 01:02:42.198935 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 01:02:42.198996 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 01:02:42.199078 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260929 01:02:42.199625 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:42.195750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52379 (local address 127.0.98.3:33501)
0929 01:02:42.195853 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:42.195858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:42.195874 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:02:42.195918 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:42.195920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:42.195960 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:42.196036 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:02:42.196331 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:42.196998 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:42.197989 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:42.198298 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:42.198392 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:42.198697 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:02:42.198791 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:02:42.198886 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:42.199450 (+ 564us) server_negotiation.cc:299] Negotiation successful
0929 01:02:42.199505 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 01:02:42.220468 7914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:42.220606 7914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:02:42.222393 7914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad71ec649f3e43ba92633d3a25a3e89c: Sending DeleteTablet for 3 replicas of tablet 2bf449bdd2d84317beaba14cbcbc6a31
I20260929 01:02:42.222882 8050 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bf449bdd2d84317beaba14cbcbc6a31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:42 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56970
I20260929 01:02:42.222925 8330 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bf449bdd2d84317beaba14cbcbc6a31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:42 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57930
I20260929 01:02:42.222957 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bf449bdd2d84317beaba14cbcbc6a31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:02:42 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48188
I20260929 01:02:42.223048 8448 tablet_replica.cc:330] T 2bf449bdd2d84317beaba14cbcbc6a31 P 48b0eb54a51d43078a7747d7edf55c8e: stopping tablet replica
I20260929 01:02:42.223098 8449 tablet_replica.cc:330] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5: stopping tablet replica
I20260929 01:02:42.223104 8450 tablet_replica.cc:330] T 2bf449bdd2d84317beaba14cbcbc6a31 P 609dfbb102b049f598299362032340ed: stopping tablet replica
I20260929 01:02:42.223155 8448 raft_consensus.cc:2229] T 2bf449bdd2d84317beaba14cbcbc6a31 P 48b0eb54a51d43078a7747d7edf55c8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:42.223191 8449 raft_consensus.cc:2229] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:02:42.223192 8450 raft_consensus.cc:2229] T 2bf449bdd2d84317beaba14cbcbc6a31 P 609dfbb102b049f598299362032340ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:02:42.223260 8448 raft_consensus.cc:2258] T 2bf449bdd2d84317beaba14cbcbc6a31 P 48b0eb54a51d43078a7747d7edf55c8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:42.223284 8450 raft_consensus.cc:2258] T 2bf449bdd2d84317beaba14cbcbc6a31 P 609dfbb102b049f598299362032340ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:42.223368 8449 raft_consensus.cc:2258] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:02:42.223695 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 7986
I20260929 01:02:42.224308 8450 ts_tablet_manager.cc:1905] T 2bf449bdd2d84317beaba14cbcbc6a31 P 609dfbb102b049f598299362032340ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:42.225680 8449 ts_tablet_manager.cc:1905] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:02:42.227111 8450 ts_tablet_manager.cc:1918] T 2bf449bdd2d84317beaba14cbcbc6a31 P 609dfbb102b049f598299362032340ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:02:42.227208 8450 log.cc:1196] T 2bf449bdd2d84317beaba14cbcbc6a31 P 609dfbb102b049f598299362032340ed: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-1/wal/wals/2bf449bdd2d84317beaba14cbcbc6a31
I20260929 01:02:42.227520 8450 ts_tablet_manager.cc:1939] T 2bf449bdd2d84317beaba14cbcbc6a31 P 609dfbb102b049f598299362032340ed: Deleting consensus metadata
I20260929 01:02:42.228155 7900 catalog_manager.cc:4841] TS 609dfbb102b049f598299362032340ed (127.0.98.2:42457): tablet 2bf449bdd2d84317beaba14cbcbc6a31 (table test-table [id=e905fa812f854923b62f1c00db1c50fc]) successfully deleted
I20260929 01:02:42.228637 8449 ts_tablet_manager.cc:1918] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 01:02:42.228741 8449 log.cc:1196] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/2bf449bdd2d84317beaba14cbcbc6a31
I20260929 01:02:42.228996 8449 ts_tablet_manager.cc:1939] T 2bf449bdd2d84317beaba14cbcbc6a31 P 41ef31ce05104786bad477cb8fd4bbd5: Deleting consensus metadata
I20260929 01:02:42.229655 7899 catalog_manager.cc:4841] TS 41ef31ce05104786bad477cb8fd4bbd5 (127.0.98.3:33501): tablet 2bf449bdd2d84317beaba14cbcbc6a31 (table test-table [id=e905fa812f854923b62f1c00db1c50fc]) successfully deleted
W20260929 01:02:42.233101 7899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38557 (error 108)
I20260929 01:02:42.233295 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 8126
I20260929 01:02:42.235178 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:42.234733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38557 (local address 127.0.0.1:56830)
0929 01:02:42.235014 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:42.235099 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":106,"threads_started":1}
W20260929 01:02:42.235307 7899 catalog_manager.cc:4584] TS 48b0eb54a51d43078a7747d7edf55c8e (127.0.98.1:38557): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2bf449bdd2d84317beaba14cbcbc6a31: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38557: connect: Connection refused (error 111)
I20260929 01:02:42.239288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 8266
I20260929 01:02:42.245931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 7882
2026-09-29T01:02:42Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3367 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:02:42 dist-test-slave-7zdl krb5kdc[8517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:02:42 dist-test-slave-7zdl krb5kdc[8517](info): set up 2 sockets
Sep 29 01:02:42 dist-test-slave-7zdl krb5kdc[8517](info): commencing operation
krb5kdc: starting...
W20260929 01:02:43.283540 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:02:43 dist-test-slave-7zdl krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:02:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:02:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:02:43.401913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36875
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:43.509089 8530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:43.509335 8530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:43.509374 8530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:43.513345 8530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:02:43.513399 8530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:43.513418 8530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:43.513434 8530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:02:43.513494 8530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:02:43.518857 8530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:43.519827 8530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:43.520838 8530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:02:43.529227 8530 server_base.cc:1034] running on GCE node
W20260929 01:02:43.529142 8536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:43.529166 8535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:43.529124 8538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:43.529963 8530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:43.531064 8530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:43.532255 8530 hybrid_clock.cc:648] HybridClock initialized: now 1790643763532237 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T01:02:43Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1292 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:02:43 dist-test-slave-7zdl krb5kdc[8543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:02:43 dist-test-slave-7zdl krb5kdc[8543](info): set up 2 sockets
Sep 29 01:02:43 dist-test-slave-7zdl krb5kdc[8543](info): commencing operation
krb5kdc: starting...
W20260929 01:02:44.579530 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:02:44 dist-test-slave-7zdl krb5kdc[8543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:02:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:02:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:02:44.696298 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36577
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:44.804234 8556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:44.804476 8556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:44.804513 8556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:44.808523 8556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:02:44.808575 8556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:44.808594 8556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:44.808610 8556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:02:44.808625 8556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:02:44.813742 8556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:44.814679 8556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:44.815616 8556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:44.823732 8564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:44.823732 8561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:44.823763 8562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:44.823900 8556 server_base.cc:1034] running on GCE node
I20260929 01:02:44.824534 8556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:44.825565 8556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:44.826726 8556 hybrid_clock.cc:648] HybridClock initialized: now 1790643764826696 us; error 42 us; skew 500 ppm
Sep 29 01:02:44 dist-test-slave-7zdl krb5kdc[8543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:44.835340 8556 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:02:44.836292 8556 webserver.cc:460] Webserver started at http://127.0.98.62:43023/ using document root <none> and password file <none>
I20260929 01:02:44.836881 8556 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:44.836971 8556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:44.837203 8556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:44.839603 8556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "30a7e1d5cd134c26a2677120823a4b7a"
format_stamp: "Formatted at 2026-09-29 01:02:44 on dist-test-slave-7zdl"
server_key: "d3d6bd8985bbbea42abcaf75f1fff7c0"
server_key_iv: "882ca75af15028a38271c3764310be80"
server_key_version: "encryptionkey@0"
I20260929 01:02:44.840181 8556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "30a7e1d5cd134c26a2677120823a4b7a"
format_stamp: "Formatted at 2026-09-29 01:02:44 on dist-test-slave-7zdl"
server_key: "d3d6bd8985bbbea42abcaf75f1fff7c0"
server_key_iv: "882ca75af15028a38271c3764310be80"
server_key_version: "encryptionkey@0"
I20260929 01:02:44.843899 8556 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:02:44.846232 8571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:44.847543 8556 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:02:44.847685 8556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "30a7e1d5cd134c26a2677120823a4b7a"
format_stamp: "Formatted at 2026-09-29 01:02:44 on dist-test-slave-7zdl"
server_key: "d3d6bd8985bbbea42abcaf75f1fff7c0"
server_key_iv: "882ca75af15028a38271c3764310be80"
server_key_version: "encryptionkey@0"
I20260929 01:02:44.847790 8556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:44.856797 8556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:44.859934 8556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:44.860113 8556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:44.866879 8556 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36577
I20260929 01:02:44.866901 8623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36577 every 8 connection(s)
I20260929 01:02:44.868119 8556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:02:44.870841 8624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:02:44.872048 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 8556
I20260929 01:02:44.872179 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:02:44.872483 392 external_mini_cluster.cc:1428] Setting key f9fc97a3af91948e0096855fdbd5ddea
I20260929 01:02:44.876474 8624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a: Bootstrap starting.
I20260929 01:02:44.878593 8624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a: Neither blocks nor log segments found. Creating new log.
I20260929 01:02:44.879361 8624 log.cc:824] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:02:44 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:44.881469 8624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a: No bootstrap required, opened a new log
I20260929 01:02:44.884496 8624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a7e1d5cd134c26a2677120823a4b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36577 } }
I20260929 01:02:44.884749 8624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:02:44.884837 8624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a7e1d5cd134c26a2677120823a4b7a, State: Initialized, Role: FOLLOWER
I20260929 01:02:44.885289 8624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a7e1d5cd134c26a2677120823a4b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36577 } }
I20260929 01:02:44.885411 8624 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:02:44.885496 8624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:02:44.885586 8624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:02:44.886513 8624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a7e1d5cd134c26a2677120823a4b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36577 } }
I20260929 01:02:44.886893 8624 leader_election.cc:302] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30a7e1d5cd134c26a2677120823a4b7a; no voters:
I20260929 01:02:44.887217 8624 leader_election.cc:288] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:02:44.887225 8627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:44.873871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36126 (local address 127.0.98.62:36577)
0929 01:02:44.874273 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:44.874287 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:44.874321 (+ 34us) server_negotiation.cc:407] Connection header received
0929 01:02:44.875003 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:44.875020 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:44.875329 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:44.875648 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:44.876411 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:44.877407 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:44.878255 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:44.878788 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:44.879016 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:44.881882 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:44.881887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:44.881900 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:44.881925 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:44.884122 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:44.884606 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:44.884610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:44.884614 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:44.884714 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:44.885029 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:44.885031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:44.885032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:44.885526 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:44.885719 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:44.886034 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 01:02:44.886312 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":111,"threads_started":1}
I20260929 01:02:44.887333 8629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:02:44.887522 8629 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [term 1 LEADER]: Becoming Leader. State: Replica: 30a7e1d5cd134c26a2677120823a4b7a, State: Running, Role: LEADER
I20260929 01:02:44.887817 8629 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a7e1d5cd134c26a2677120823a4b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36577 } }
I20260929 01:02:44.888520 8624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:02:44.890000 8630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30a7e1d5cd134c26a2677120823a4b7a. Latest consensus state: current_term: 1 leader_uuid: "30a7e1d5cd134c26a2677120823a4b7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a7e1d5cd134c26a2677120823a4b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36577 } } }
I20260929 01:02:44.890121 8630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:44.890393 8631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30a7e1d5cd134c26a2677120823a4b7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a7e1d5cd134c26a2677120823a4b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36577 } } }
I20260929 01:02:44.890478 8631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:44.890952 8641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:02:44.893517 8641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:02:44.898756 8641 catalog_manager.cc:1347] Generated new cluster ID: 3d45100313b74548a689cde500321df8
I20260929 01:02:44.898974 8641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:02:44.906443 8641 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:02:44.907325 8641 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:02:44.915918 8641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a: Generated new TSK 0
I20260929 01:02:44.916718 8641 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 01:02:44.953946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36577
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:45.062656 8650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:45.062915 8650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:45.062956 8650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:45.066823 8650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:45.066869 8650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:45.066926 8650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:02:45.071910 8650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:45.072795 8650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:45.073694 8650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:45.082746 8656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:45.082782 8650 server_base.cc:1034] running on GCE node
W20260929 01:02:45.082837 8658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:45.082975 8655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:45.083331 8650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:45.083863 8650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:45.085052 8650 hybrid_clock.cc:648] HybridClock initialized: now 1790643765085020 us; error 46 us; skew 500 ppm
Sep 29 01:02:45 dist-test-slave-7zdl krb5kdc[8543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:45.087744 8650 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:02:45.088531 8650 webserver.cc:460] Webserver started at http://127.0.98.1:40237/ using document root <none> and password file <none>
I20260929 01:02:45.089190 8650 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:45.089274 8650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:45.089480 8650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:45.091552 8650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "c4c2c35c6f524dffa1ccb7d0f75d8bbe"
format_stamp: "Formatted at 2026-09-29 01:02:45 on dist-test-slave-7zdl"
server_key: "c89f0cc183be32fb8df57b971b3f6040"
server_key_iv: "c010f0b4d5ee489b34305e7ccf801ac1"
server_key_version: "encryptionkey@0"
I20260929 01:02:45.092089 8650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c4c2c35c6f524dffa1ccb7d0f75d8bbe"
format_stamp: "Formatted at 2026-09-29 01:02:45 on dist-test-slave-7zdl"
server_key: "c89f0cc183be32fb8df57b971b3f6040"
server_key_iv: "c010f0b4d5ee489b34305e7ccf801ac1"
server_key_version: "encryptionkey@0"
I20260929 01:02:45.095543 8650 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:02:45.097718 8665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:45.098922 8650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:02:45.099071 8650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "c4c2c35c6f524dffa1ccb7d0f75d8bbe"
format_stamp: "Formatted at 2026-09-29 01:02:45 on dist-test-slave-7zdl"
server_key: "c89f0cc183be32fb8df57b971b3f6040"
server_key_iv: "c010f0b4d5ee489b34305e7ccf801ac1"
server_key_version: "encryptionkey@0"
I20260929 01:02:45.099171 8650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:45.114192 8650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:45.117437 8650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:45.117642 8650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:45.118243 8650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:45.119218 8650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:45.119282 8650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:45.119346 8650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:45.119400 8650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:45.128322 8650 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41151
I20260929 01:02:45.128355 8778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41151 every 8 connection(s)
I20260929 01:02:45.129331 8650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
Sep 29 01:02:45 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:45.139626 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 8650
I20260929 01:02:45.139761 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:02:45.140048 392 external_mini_cluster.cc:1428] Setting key e2b526eba99418d1a7df51bd31154a6a
I20260929 01:02:45.142849 8627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:45.131203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47943 (local address 127.0.98.62:36577)
0929 01:02:45.131365 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:45.131370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:45.132197 (+ 827us) server_negotiation.cc:407] Connection header received
0929 01:02:45.133491 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:45.133493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:45.133537 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:45.133611 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:45.134763 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:45.135359 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:45.136416 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:45.136633 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:45.136706 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:45.139022 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:45.139025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:45.139027 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:45.139049 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:45.140723 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:45.141205 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:45.141210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:45.141212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:45.141294 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:45.141647 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:45.141650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:45.141652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:45.141886 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:45.141989 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:45.142428 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 01:02:45.142580 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 01:02:45.143501 8781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:45.131626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.1:47943)
0929 01:02:45.132052 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:45.132093 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:45.133310 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:45.133785 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:45.133791 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:45.134141 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:45.134642 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:45.134654 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:45.135460 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:45.135462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:45.136319 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:45.136327 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:45.136751 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:45.136753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:45.136886 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:45.137425 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:45.137440 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:45.138876 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:45.140835 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:45.140839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:45.140849 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:45.141095 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:45.141411 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:45.141415 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:45.141417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:45.141550 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:45.141987 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:45.141992 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:45.142199 (+ 207us) client_negotiation.cc:769] Sending connection context
0929 01:02:45.142361 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 01:02:45.142597 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":108,"threads_started":1}
I20260929 01:02:45.144613 8779 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36577
I20260929 01:02:45.144824 8779 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:45.145272 8779 heartbeater.cc:511] Master 127.0.98.62:36577 requested a full tablet report, sending...
I20260929 01:02:45.146675 8588 ts_manager.cc:194] Registered new tserver with Master: c4c2c35c6f524dffa1ccb7d0f75d8bbe (127.0.98.1:41151)
I20260929 01:02:45.147800 8588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47943
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 01:02:45.166563 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36577
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:45.276947 8784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:45.277208 8784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:45.277249 8784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:45.281275 8784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:45.281332 8784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:45.281386 8784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:02:45.286461 8784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:45.287498 8784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:45.288578 8784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:45.298266 8789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:45.298249 8790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:45.298312 8792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:45.298424 8784 server_base.cc:1034] running on GCE node
I20260929 01:02:45.298980 8784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:45.299583 8784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:45.300768 8784 hybrid_clock.cc:648] HybridClock initialized: now 1790643765300749 us; error 30 us; skew 500 ppm
Sep 29 01:02:45 dist-test-slave-7zdl krb5kdc[8543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:45.303537 8784 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:02:45.304375 8784 webserver.cc:460] Webserver started at http://127.0.98.2:35209/ using document root <none> and password file <none>
I20260929 01:02:45.304929 8784 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:45.305017 8784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:45.305248 8784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:45.307335 8784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "3babfd05d77743b6912e62741d7aba8d"
format_stamp: "Formatted at 2026-09-29 01:02:45 on dist-test-slave-7zdl"
server_key: "6c912540544d6535a85bec359e8f8a9b"
server_key_iv: "cd6151bf31878afa4eb5d4192d303ea5"
server_key_version: "encryptionkey@0"
I20260929 01:02:45.307883 8784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3babfd05d77743b6912e62741d7aba8d"
format_stamp: "Formatted at 2026-09-29 01:02:45 on dist-test-slave-7zdl"
server_key: "6c912540544d6535a85bec359e8f8a9b"
server_key_iv: "cd6151bf31878afa4eb5d4192d303ea5"
server_key_version: "encryptionkey@0"
I20260929 01:02:45.311439 8784 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:02:45.313797 8799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:45.315156 8784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:02:45.315307 8784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "3babfd05d77743b6912e62741d7aba8d"
format_stamp: "Formatted at 2026-09-29 01:02:45 on dist-test-slave-7zdl"
server_key: "6c912540544d6535a85bec359e8f8a9b"
server_key_iv: "cd6151bf31878afa4eb5d4192d303ea5"
server_key_version: "encryptionkey@0"
I20260929 01:02:45.315418 8784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:45.328653 8784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:45.331274 8784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:45.331492 8784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:45.332033 8784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:45.332804 8784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:45.332875 8784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:45.332943 8784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:45.332989 8784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:45.341377 8784 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42223
I20260929 01:02:45.341410 8912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42223 every 8 connection(s)
I20260929 01:02:45.342217 8784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 01:02:45 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:45.352409 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 8784
I20260929 01:02:45.352548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:02:45.352857 392 external_mini_cluster.cc:1428] Setting key 46bb0f6a7e674f1f8271c61fb4a5a0b1
I20260929 01:02:45.356030 8627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:45.343971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34175 (local address 127.0.98.62:36577)
0929 01:02:45.344106 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:45.344110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:45.344913 (+ 803us) server_negotiation.cc:407] Connection header received
0929 01:02:45.346306 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:45.346310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:45.346391 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:45.346483 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:45.347632 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:45.348164 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:45.349255 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:45.349500 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:45.349611 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:45.352123 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:45.352126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:45.352128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:45.352151 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:45.353790 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:45.354342 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:45.354346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:45.354348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:45.354441 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:45.354755 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:45.354761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:45.354765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:45.355069 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:45.355171 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:45.355636 (+ 465us) server_negotiation.cc:299] Negotiation successful
0929 01:02:45.355768 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 01:02:45.357012 8915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:45.344343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.2:34175)
0929 01:02:45.344765 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:45.344804 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:45.346099 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:45.346643 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:45.346649 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:45.347049 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:45.347492 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:45.347504 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:45.348305 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:45.348307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:45.349133 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:45.349141 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:45.349643 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:45.349647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:45.349814 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:45.350399 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:45.350415 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:45.351942 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:45.353904 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:45.353909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:45.353922 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:45.354239 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:45.354535 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:45.354537 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:45.354539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:45.354660 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:45.355178 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:45.355182 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:45.355400 (+ 218us) client_negotiation.cc:769] Sending connection context
0929 01:02:45.355562 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 01:02:45.355792 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":113,"threads_started":1}
I20260929 01:02:45.358140 8913 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36577
I20260929 01:02:45.358376 8913 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:45.358951 8913 heartbeater.cc:511] Master 127.0.98.62:36577 requested a full tablet report, sending...
I20260929 01:02:45.359982 8588 ts_manager.cc:194] Registered new tserver with Master: 3babfd05d77743b6912e62741d7aba8d (127.0.98.2:42223)
I20260929 01:02:45.360548 8588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34175
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 01:02:45.382313 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36577
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:45.493747 8918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:45.493978 8918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:45.494023 8918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:45.498715 8918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:45.498770 8918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:45.498881 8918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:02:45.504086 8918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:45.505031 8918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:45.505954 8918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:45.515398 8924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:45.515409 8926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:45.515453 8923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:45.515648 8918 server_base.cc:1034] running on GCE node
I20260929 01:02:45.516193 8918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:45.516861 8918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:45.518042 8918 hybrid_clock.cc:648] HybridClock initialized: now 1790643765518027 us; error 30 us; skew 500 ppm
Sep 29 01:02:45 dist-test-slave-7zdl krb5kdc[8543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:45.520782 8918 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:02:45.521797 8918 webserver.cc:460] Webserver started at http://127.0.98.3:36291/ using document root <none> and password file <none>
I20260929 01:02:45.522372 8918 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:45.522462 8918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:45.522691 8918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:02:45.524799 8918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "f1a57b5d42174b0b82d809129b6f896c"
format_stamp: "Formatted at 2026-09-29 01:02:45 on dist-test-slave-7zdl"
server_key: "f7ec0fa19434db3f5ea4031db5685ab4"
server_key_iv: "4e9d9cb43dc2564fd6affbfd1c3cd6ff"
server_key_version: "encryptionkey@0"
I20260929 01:02:45.525354 8918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f1a57b5d42174b0b82d809129b6f896c"
format_stamp: "Formatted at 2026-09-29 01:02:45 on dist-test-slave-7zdl"
server_key: "f7ec0fa19434db3f5ea4031db5685ab4"
server_key_iv: "4e9d9cb43dc2564fd6affbfd1c3cd6ff"
server_key_version: "encryptionkey@0"
I20260929 01:02:45.529016 8918 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:02:45.531477 8933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:45.532645 8918 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:02:45.532778 8918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "f1a57b5d42174b0b82d809129b6f896c"
format_stamp: "Formatted at 2026-09-29 01:02:45 on dist-test-slave-7zdl"
server_key: "f7ec0fa19434db3f5ea4031db5685ab4"
server_key_iv: "4e9d9cb43dc2564fd6affbfd1c3cd6ff"
server_key_version: "encryptionkey@0"
I20260929 01:02:45.532886 8918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:45.549211 8918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:45.552552 8918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:45.552762 8918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:45.553342 8918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:45.554147 8918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:45.554224 8918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:45.554301 8918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:45.554343 8918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:45.562652 8918 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39879
I20260929 01:02:45.562717 9046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39879 every 8 connection(s)
I20260929 01:02:45.563550 8918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:02:45.568252 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 8918
I20260929 01:02:45.568387 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:02:45.568653 392 external_mini_cluster.cc:1428] Setting key ddc6258bbe1ef115748e29379f42709e
Sep 29 01:02:45 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:02:45.576737 8627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:45.565217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48675 (local address 127.0.98.62:36577)
0929 01:02:45.565359 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:45.565363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:45.566127 (+ 764us) server_negotiation.cc:407] Connection header received
0929 01:02:45.567534 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:45.567537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:45.567595 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:45.567703 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:45.568927 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:45.569654 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:45.571005 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:45.571235 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:45.571351 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:45.573566 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:45.573570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:45.573573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:45.573605 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:45.574763 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:45.575296 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:45.575301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:45.575303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:45.575384 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:45.575704 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:45.575708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:45.575710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:45.575878 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:45.575997 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:45.576462 (+ 465us) server_negotiation.cc:299] Negotiation successful
0929 01:02:45.576574 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 01:02:45.577486 9049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:45.565581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.3:48675)
0929 01:02:45.565992 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:45.566031 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:45.567331 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:45.567869 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:45.567876 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:45.568224 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:45.568782 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:45.568797 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:45.569769 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:45.569772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:45.570892 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:45.570901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:45.571376 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:45.571380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:45.571551 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:45.572044 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:45.572060 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:45.573415 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:02:45.574923 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:45.574929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:45.574942 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:45.575196 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:45.575478 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:02:45.575481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:02:45.575483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:02:45.575614 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:02:45.576006 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:02:45.576014 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:02:45.576235 (+ 221us) client_negotiation.cc:769] Sending connection context
0929 01:02:45.576408 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 01:02:45.576713 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":107,"threads_started":1}
I20260929 01:02:45.578565 9047 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36577
I20260929 01:02:45.578863 9047 heartbeater.cc:464] Registering TS with master...
I20260929 01:02:45.579495 9047 heartbeater.cc:511] Master 127.0.98.62:36577 requested a full tablet report, sending...
I20260929 01:02:45.580556 8588 ts_manager.cc:194] Registered new tserver with Master: f1a57b5d42174b0b82d809129b6f896c (127.0.98.3:39879)
I20260929 01:02:45.581264 8588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48675
I20260929 01:02:45.582258 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 01:02:45.652988 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 8650
I20260929 01:02:45.658380 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 8784
I20260929 01:02:45.663453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 8918
I20260929 01:02:45.668357 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 8556
I20260929 01:02:45.674708 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36577
--webserver_interface=127.0.98.62
--webserver_port=43023
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:02:45.786321 9056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:45.786629 9056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:45.786698 9056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:45.790808 9056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:02:45.790894 9056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:45.790931 9056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:45.790987 9056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:02:45.791023 9056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:02:45.796267 9056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:45.797299 9056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:45.798344 9056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:45.806957 9062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:45.806950 9061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:45.806973 9064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:45.807156 9056 server_base.cc:1034] running on GCE node
I20260929 01:02:45.807710 9056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:45.808681 9056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:45.809851 9056 hybrid_clock.cc:648] HybridClock initialized: now 1790643765809829 us; error 31 us; skew 500 ppm
Sep 29 01:02:45 dist-test-slave-7zdl krb5kdc[8543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:45.812536 9056 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:02:45.813398 9056 webserver.cc:460] Webserver started at http://127.0.98.62:43023/ using document root <none> and password file <none>
I20260929 01:02:45.813942 9056 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:45.814057 9056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:45.817677 9056 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:45.819788 9071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:45.820956 9056 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:45.821089 9056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "30a7e1d5cd134c26a2677120823a4b7a"
format_stamp: "Formatted at 2026-09-29 01:02:44 on dist-test-slave-7zdl"
server_key: "d3d6bd8985bbbea42abcaf75f1fff7c0"
server_key_iv: "882ca75af15028a38271c3764310be80"
server_key_version: "encryptionkey@0"
I20260929 01:02:45.821671 9056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:45.833321 9056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:45.836043 9056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:45.836237 9056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:45.843214 9056 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36577
I20260929 01:02:45.843232 9123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36577 every 8 connection(s)
I20260929 01:02:45.844238 9056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:02:45.848315 9124 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:02:45.849859 9124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a: Bootstrap starting.
I20260929 01:02:45.850589 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 9056
I20260929 01:02:45.851055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41151
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36577
--webserver_port=40237
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:02:45.866240 9124 log.cc:824] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a: Log is configured to *not* fsync() on all Append() calls
I20260929 01:02:45.873018 9124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:02:45.873380 9124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a: Bootstrap complete.
I20260929 01:02:45.877507 9124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a7e1d5cd134c26a2677120823a4b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36577 } }
I20260929 01:02:45.877935 9124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a7e1d5cd134c26a2677120823a4b7a, State: Initialized, Role: FOLLOWER
I20260929 01:02:45.878418 9124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a7e1d5cd134c26a2677120823a4b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36577 } }
I20260929 01:02:45.878561 9124 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:02:45.878646 9124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:02:45.878762 9124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:02:45.880610 9124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a7e1d5cd134c26a2677120823a4b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36577 } }
I20260929 01:02:45.880946 9124 leader_election.cc:302] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30a7e1d5cd134c26a2677120823a4b7a; no voters:
I20260929 01:02:45.881311 9124 leader_election.cc:288] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 01:02:45.881366 9128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:02:45.881584 9128 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [term 2 LEADER]: Becoming Leader. State: Replica: 30a7e1d5cd134c26a2677120823a4b7a, State: Running, Role: LEADER
I20260929 01:02:45.881919 9128 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a7e1d5cd134c26a2677120823a4b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36577 } }
I20260929 01:02:45.882516 9124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:02:45.882899 9130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30a7e1d5cd134c26a2677120823a4b7a. Latest consensus state: current_term: 2 leader_uuid: "30a7e1d5cd134c26a2677120823a4b7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a7e1d5cd134c26a2677120823a4b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36577 } } }
I20260929 01:02:45.883085 9130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:45.883265 9129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "30a7e1d5cd134c26a2677120823a4b7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a7e1d5cd134c26a2677120823a4b7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36577 } } }
I20260929 01:02:45.883379 9129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a [sys.catalog]: This master's current role is: LEADER
I20260929 01:02:45.883669 9133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:02:45.887009 9133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:02:45.887673 9133 catalog_manager.cc:1259] Loaded cluster ID: 3d45100313b74548a689cde500321df8
I20260929 01:02:45.887737 9133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:02:45.890034 9133 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:02:45.890682 9133 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 30a7e1d5cd134c26a2677120823a4b7a: Loaded TSK: 0
I20260929 01:02:45.891896 9133 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 01:02:45.961743 9126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:45.962009 9126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:45.962083 9126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:45.966109 9126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:45.966181 9126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:45.966262 9126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:02:45.971352 9126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41151
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40237
--tserver_master_addrs=127.0.98.62:36577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:45.972287 9126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:45.973229 9126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:45.983536 9153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:45.983542 9150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:45.983549 9151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:45.983826 9126 server_base.cc:1034] running on GCE node
I20260929 01:02:45.984460 9126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:45.985239 9126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:45.986461 9126 hybrid_clock.cc:648] HybridClock initialized: now 1790643765986428 us; error 50 us; skew 500 ppm
Sep 29 01:02:45 dist-test-slave-7zdl krb5kdc[8543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643765, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:45.989868 9126 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 01:02:45.990998 9126 webserver.cc:460] Webserver started at http://127.0.98.1:40237/ using document root <none> and password file <none>
I20260929 01:02:45.991706 9126 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:45.991815 9126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:45.996750 9126 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 01:02:45.999228 9160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:46.000697 9126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:02:46.000823 9126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "c4c2c35c6f524dffa1ccb7d0f75d8bbe"
format_stamp: "Formatted at 2026-09-29 01:02:45 on dist-test-slave-7zdl"
server_key: "c89f0cc183be32fb8df57b971b3f6040"
server_key_iv: "c010f0b4d5ee489b34305e7ccf801ac1"
server_key_version: "encryptionkey@0"
I20260929 01:02:46.001488 9126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:46.029059 9126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:46.032554 9126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:46.032747 9126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:46.033406 9126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:46.034332 9126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:46.034386 9126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:46.034435 9126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:46.034464 9126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:46.044199 9126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41151
I20260929 01:02:46.044222 9273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41151 every 8 connection(s)
I20260929 01:02:46.045217 9126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:02:46.047201 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 9126
I20260929 01:02:46.047663 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42223
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36577
--webserver_port=35209
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:02:46 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:46 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:46.060034 9277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:46.047425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.1:33125)
0929 01:02:46.047942 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:46.047991 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:46.049694 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:46.050852 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:46.050864 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:46.051377 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:46.052108 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:46.052132 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.053346 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.053352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:46.055017 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:46.055031 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.055772 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.055779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:46.055996 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:46.056638 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:46.056663 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:46.059319 (+ 2656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":123,"threads_started":1}
W20260929 01:02:46.060550 9274 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36577 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36577: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260929 01:02:46.060962 9276 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:46.047321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33125 (local address 127.0.98.62:36577)
0929 01:02:46.048046 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:46.048068 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:46.048117 (+ 49us) server_negotiation.cc:407] Connection header received
0929 01:02:46.050130 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:46.050141 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:46.050426 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:46.050685 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:46.052292 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.053204 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.055142 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.055595 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.055712 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:46.059593 (+ 3881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33125: BlockingRecv error: recv got EOF from 127.0.98.1:33125 (error 108)
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":121,"threads_started":1}
Sep 29 01:02:46 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:46 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:46.066082 9277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:46.061012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.1:55665)
0929 01:02:46.061361 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:46.061375 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:46.061479 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:46.061751 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:46.061754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:46.061907 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:46.061988 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:46.061994 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.062712 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.062716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:46.063289 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:46.063297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.063733 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.063736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:46.063827 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:46.064352 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:46.064363 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:46.065950 (+ 1587us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268}
W20260929 01:02:46.066349 9279 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:46.061063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55665 (local address 127.0.98.62:36577)
0929 01:02:46.061330 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:46.061339 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:46.061442 (+ 103us) server_negotiation.cc:407] Connection header received
0929 01:02:46.061584 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:46.061587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:46.061657 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:46.061740 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:46.062097 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.062605 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.063407 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.063631 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.063733 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:46.066056 (+ 2323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55665: BlockingRecv error: recv got EOF from 127.0.98.1:55665 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":113,"threads_started":1}
Sep 29 01:02:46 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:46 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:46.071415 9277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:46.066517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.1:33147)
0929 01:02:46.066619 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:46.066632 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:46.066708 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:46.066966 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:46.066969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:46.067115 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:46.067195 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:46.067200 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.067830 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.067833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:46.068645 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:46.068656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.069042 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.069046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:46.069124 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:46.069523 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:46.069541 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:46.071305 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 01:02:46.071501 9279 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:46.066604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33147 (local address 127.0.98.62:36577)
0929 01:02:46.066694 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:46.066697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:46.066707 (+ 10us) server_negotiation.cc:407] Connection header received
0929 01:02:46.066784 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:46.066786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:46.066862 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:46.066928 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:46.067298 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.067737 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.068759 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.068948 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.069015 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:46.071406 (+ 2391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33147: BlockingRecv error: recv got EOF from 127.0.98.1:33147 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260929 01:02:46.071657 9274 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 01:02:46.159559 9278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:46.159796 9278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:46.159837 9278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:46.163637 9278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:46.163686 9278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:46.163740 9278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:02:46.168819 9278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35209
--tserver_master_addrs=127.0.98.62:36577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:46.169723 9278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:46.170637 9278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:46.180153 9284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:46.180166 9285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:46.180505 9278 server_base.cc:1034] running on GCE node
W20260929 01:02:46.180166 9287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:46.180974 9278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:46.181665 9278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:46.182885 9278 hybrid_clock.cc:648] HybridClock initialized: now 1790643766182861 us; error 30 us; skew 500 ppm
Sep 29 01:02:46 dist-test-slave-7zdl krb5kdc[8543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643766, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:46.185694 9278 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 01:02:46.186527 9278 webserver.cc:460] Webserver started at http://127.0.98.2:35209/ using document root <none> and password file <none>
I20260929 01:02:46.187155 9278 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:46.187258 9278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:46.191015 9278 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 01:02:46.193020 9294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:46.194244 9278 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:02:46.194373 9278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "3babfd05d77743b6912e62741d7aba8d"
format_stamp: "Formatted at 2026-09-29 01:02:45 on dist-test-slave-7zdl"
server_key: "6c912540544d6535a85bec359e8f8a9b"
server_key_iv: "cd6151bf31878afa4eb5d4192d303ea5"
server_key_version: "encryptionkey@0"
I20260929 01:02:46.194954 9278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:46.241334 9278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:46.244392 9278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:46.244596 9278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:46.245193 9278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:46.246006 9278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:46.246086 9278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:46.246176 9278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:46.246227 9278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:46.255023 9278 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42223
I20260929 01:02:46.255049 9407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42223 every 8 connection(s)
I20260929 01:02:46.255980 9278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:02:46.264706 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 9278
Sep 29 01:02:46 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:46.265175 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39879
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36577
--webserver_port=36291
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:02:46 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 01:02:46.267253 9279 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:46.257784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54727 (local address 127.0.98.62:36577)
0929 01:02:46.257915 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:46.257919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:46.258721 (+ 802us) server_negotiation.cc:407] Connection header received
0929 01:02:46.260141 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:46.260144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:46.260210 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:46.260314 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:46.261479 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.261998 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.263178 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.263406 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.263517 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:46.267101 (+ 3584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54727: BlockingRecv error: recv got EOF from 127.0.98.2:54727 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 01:02:46.267714 9410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:46.258152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.2:54727)
0929 01:02:46.258583 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:46.258623 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:46.259925 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:46.260528 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:46.260535 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:46.260866 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:46.261330 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:46.261343 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.262123 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.262126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:46.263060 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:46.263069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.263550 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.263554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:46.263715 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:46.264227 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:46.264242 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:46.266917 (+ 2675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":113,"threads_started":1}
W20260929 01:02:46.268244 9408 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36577 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36577: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 29 01:02:46 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:46 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:46.274291 9410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:46.268708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.2:37451)
0929 01:02:46.268819 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:46.268834 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:46.268912 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:46.269188 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:46.269191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:46.269358 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:46.269448 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:46.269453 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.270579 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.270581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:46.271170 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:46.271178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.271810 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.271812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:46.271879 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:46.272358 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:46.272369 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:46.274199 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 01:02:46.274550 9279 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:46.268792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37451 (local address 127.0.98.62:36577)
0929 01:02:46.268923 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:46.268930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:46.268946 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:02:46.268989 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:46.268995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:46.269051 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:46.269145 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:46.269569 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.270474 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.271277 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.271700 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.271827 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:46.274386 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37451: BlockingRecv error: recv got EOF from 127.0.98.2:37451 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 01:02:46 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:46 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:46.279824 9410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:46.274950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.2:49543)
0929 01:02:46.275038 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:46.275054 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:46.275125 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:46.275395 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:46.275397 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:46.275511 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:46.275574 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:46.275578 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.276631 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.276634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:46.277120 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:46.277126 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.277645 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.277647 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:46.277698 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:46.278143 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:46.278154 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:46.279743 (+ 1589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 01:02:46.279979 9279 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:46.275024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49543 (local address 127.0.98.62:36577)
0929 01:02:46.275162 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:46.275168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:46.275183 (+ 15us) server_negotiation.cc:407] Connection header received
0929 01:02:46.275223 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:46.275228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:46.275280 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:46.275380 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:46.275679 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.276526 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.277217 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.277548 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.277643 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:46.279878 (+ 2235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49543: BlockingRecv error: recv got EOF from 127.0.98.2:49543 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 01:02:46.280092 9408 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 01:02:46.376677 9411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:02:46.376929 9411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:02:46.377002 9411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:02:46.381009 9411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:02:46.381065 9411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:02:46.381145 9411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:02:46.386628 9411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36291
--tserver_master_addrs=127.0.98.62:36577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:02:46.387660 9411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:02:46.388646 9411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:02:46.398160 9416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:46.398227 9411 server_base.cc:1034] running on GCE node
W20260929 01:02:46.398167 9417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:02:46.398167 9419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:02:46.398851 9411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:02:46.399421 9411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:02:46.400605 9411 hybrid_clock.cc:648] HybridClock initialized: now 1790643766400597 us; error 31 us; skew 500 ppm
Sep 29 01:02:46 dist-test-slave-7zdl krb5kdc[8543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643766, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:02:46.403173 9411 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 01:02:46.404045 9411 webserver.cc:460] Webserver started at http://127.0.98.3:36291/ using document root <none> and password file <none>
I20260929 01:02:46.404598 9411 fs_manager.cc:359] Metadata directory not provided
I20260929 01:02:46.404698 9411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:02:46.408493 9411 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260929 01:02:46.410454 9426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:46.411669 9411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 01:02:46.411798 9411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "f1a57b5d42174b0b82d809129b6f896c"
format_stamp: "Formatted at 2026-09-29 01:02:45 on dist-test-slave-7zdl"
server_key: "f7ec0fa19434db3f5ea4031db5685ab4"
server_key_iv: "4e9d9cb43dc2564fd6affbfd1c3cd6ff"
server_key_version: "encryptionkey@0"
I20260929 01:02:46.412312 9411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:02:46.434176 9411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:02:46.437040 9411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:02:46.437239 9411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:02:46.437817 9411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:02:46.438635 9411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:02:46.438704 9411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:46.438778 9411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:02:46.438843 9411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:02:46.447603 9411 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39879
I20260929 01:02:46.447630 9539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39879 every 8 connection(s)
I20260929 01:02:46.448549 9411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:02:46.451978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 9411
Sep 29 01:02:46 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:46 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 01:02:46.460580 9279 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:46.450242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33987 (local address 127.0.98.62:36577)
0929 01:02:46.450418 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:46.450423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:46.451248 (+ 825us) server_negotiation.cc:407] Connection header received
0929 01:02:46.452707 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:46.452711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:46.452765 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:46.452855 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:46.454356 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.455043 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.456456 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.456639 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.456744 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:46.460428 (+ 3684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33987: BlockingRecv error: recv got EOF from 127.0.98.3:33987 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 01:02:46.461198 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:46.450646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.3:33987)
0929 01:02:46.451109 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:46.451148 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:46.452482 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:46.453072 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:46.453084 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:46.453601 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:46.454237 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:46.454252 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.455148 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.455152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:46.456351 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:46.456361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.456794 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.456799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:46.456947 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:46.457465 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:46.457480 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:46.460303 (+ 2823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":172,"threads_started":1}
W20260929 01:02:46.461804 9540 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36577 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36577: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260929 01:02:46.461992 9276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:46.452921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36142 (local address 127.0.98.62:36577)
0929 01:02:46.453025 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:46.453027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:46.453037 (+ 10us) server_negotiation.cc:407] Connection header received
0929 01:02:46.453125 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:46.453127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:46.453160 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:46.453230 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:46.454086 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.454848 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.456016 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.456221 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.456293 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:46.457879 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:02:46.457888 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:02:46.457923 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:02:46.457958 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:02:46.459779 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:46.460291 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:46.460297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:46.460303 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:46.460451 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:02:46.460818 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:02:46.460821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:02:46.460823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:02:46.461170 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:02:46.461346 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 01:02:46.461625 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 01:02:46.461844 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 01:02:46 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:46 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:46.466970 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:46.462262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.3:35707)
0929 01:02:46.462441 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:46.462451 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:46.462543 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:46.462752 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:46.462754 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:46.462933 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:46.463037 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:46.463042 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.463757 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.463760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:46.464387 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:46.464395 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.464853 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.464855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:46.464919 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:46.465273 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:46.465283 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:46.466881 (+ 1598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 01:02:46.467286 9276 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:46.462296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35707 (local address 127.0.98.62:36577)
0929 01:02:46.462386 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:46.462390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:46.462508 (+ 118us) server_negotiation.cc:407] Connection header received
0929 01:02:46.462616 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:46.462619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:46.462653 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:46.462740 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:46.463145 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.463647 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.464529 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.464762 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.464836 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:46.466969 (+ 2133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35707: BlockingRecv error: recv got EOF from 127.0.98.3:35707 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 29 01:02:46 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:46 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:46.471575 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:46.467429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.3:39995)
0929 01:02:46.467561 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:46.467592 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:46.467660 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:46.467825 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:46.467826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:46.467935 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:46.467992 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:46.467996 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.468615 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.468617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:46.469112 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:46.469119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.469483 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.469486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:46.469535 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:46.469875 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:46.469885 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:46.471455 (+ 1570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":79}
W20260929 01:02:46.471649 9276 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:46.467469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39995 (local address 127.0.98.62:36577)
0929 01:02:46.467603 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:46.467606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:46.467632 (+ 26us) server_negotiation.cc:407] Connection header received
0929 01:02:46.467724 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:46.467726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:46.467756 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:46.467811 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:46.468080 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.468531 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.469200 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:46.469397 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:46.469465 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:46.471547 (+ 2082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39995: BlockingRecv error: recv got EOF from 127.0.98.3:39995 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 01:02:46.471817 9540 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 01:02:47 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:47 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:47.077670 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:47.072303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.1:51479)
0929 01:02:47.072594 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:47.072612 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:47.072704 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:47.072937 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:47.072939 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:47.073107 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:47.073197 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:47.073202 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:47.073889 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:47.073891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:47.074460 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:47.074467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:47.074971 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:47.074974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:47.075064 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:47.075680 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:47.075692 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:47.077554 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":86,"threads_started":1}
W20260929 01:02:47.077816 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:47.072455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51479 (local address 127.0.98.62:36577)
0929 01:02:47.072714 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:47.072718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:47.072751 (+ 33us) server_negotiation.cc:407] Connection header received
0929 01:02:47.072786 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:47.072787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:47.072835 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:47.072912 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:47.073296 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:47.073802 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:47.074557 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:47.074845 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:47.074968 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:47.077685 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51479: BlockingRecv error: recv got EOF from 127.0.98.1:51479 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
Sep 29 01:02:47 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:47 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:47.286875 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:47.280806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.2:36315)
0929 01:02:47.281130 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:47.281144 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:47.281245 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:47.281502 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:47.281504 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:47.281677 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:47.281797 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:47.281802 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:47.282614 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:47.282618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:47.283553 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:47.283565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:47.284147 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:47.284150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:47.284215 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:47.284818 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:47.284835 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:47.286659 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":90,"threads_started":1}
W20260929 01:02:47.286923 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:47.280889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36315 (local address 127.0.98.62:36577)
0929 01:02:47.281046 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:47.281049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:47.281217 (+ 168us) server_negotiation.cc:407] Connection header received
0929 01:02:47.281342 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:47.281345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:47.281392 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:47.281466 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:47.281903 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:47.282490 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:47.283680 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:47.284020 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:47.284124 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:47.286777 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36315: BlockingRecv error: recv got EOF from 127.0.98.2:36315 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 01:02:47 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:47 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:47.478189 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:47.472435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.3:52435)
0929 01:02:47.472695 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:47.472708 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:47.472806 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:47.473107 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:47.473109 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:47.473290 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:47.473383 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:47.473389 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:47.474524 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:47.474526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:47.475250 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:47.475259 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:47.475656 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:47.475658 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:47.475717 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:47.476189 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:47.476200 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:47.478011 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":94,"threads_started":1}
W20260929 01:02:47.478224 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:47.472495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52435 (local address 127.0.98.62:36577)
0929 01:02:47.472616 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:47.472620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:47.472775 (+ 155us) server_negotiation.cc:407] Connection header received
0929 01:02:47.472902 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:47.472904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:47.472950 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:47.473023 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:47.473531 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:47.474409 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:47.475354 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:47.475562 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:47.475639 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:47.478119 (+ 2480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52435: BlockingRecv error: recv got EOF from 127.0.98.3:52435 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 01:02:48 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:48 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:48.084455 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:48.078590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.1:52667)
0929 01:02:48.078999 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:48.079012 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:48.079111 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:48.079352 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:48.079354 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:48.079525 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:48.079618 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:48.079623 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:48.080370 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:48.080374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:48.081164 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:48.081176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:48.081726 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:48.081729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:48.081791 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:48.082318 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:48.082329 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:48.084306 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":89,"threads_started":1}
W20260929 01:02:48.084570 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:48.078655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52667 (local address 127.0.98.62:36577)
0929 01:02:48.078935 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:48.078938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:48.079082 (+ 144us) server_negotiation.cc:407] Connection header received
0929 01:02:48.079186 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:48.079188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:48.079230 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:48.079305 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:48.079709 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:48.080253 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:48.081304 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:48.081629 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:48.081708 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:48.084428 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52667: BlockingRecv error: recv got EOF from 127.0.98.1:52667 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":59,"threads_started":1}
Sep 29 01:02:48 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:48 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:48.293140 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:48.287732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.2:49981)
0929 01:02:48.288050 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:48.288062 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:48.288187 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:48.288462 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:48.288465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:48.288664 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:48.288754 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:48.288760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:48.289502 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:48.289509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:48.290067 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:48.290074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:48.290520 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:48.290523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:48.290607 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:48.291161 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:48.291173 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:48.292947 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":80,"threads_started":1}
W20260929 01:02:48.293164 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:48.287810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49981 (local address 127.0.98.62:36577)
0929 01:02:48.287967 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:48.287972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:48.288138 (+ 166us) server_negotiation.cc:407] Connection header received
0929 01:02:48.288295 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:48.288299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:48.288355 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:48.288445 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:48.288872 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:48.289395 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:48.290184 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:48.290409 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:48.290519 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:48.293066 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49981: BlockingRecv error: recv got EOF from 127.0.98.2:49981 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 01:02:48 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:48 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:48.484462 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:48.479056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.3:43805)
0929 01:02:48.479324 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:48.479338 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:48.479423 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:48.479657 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:48.479659 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:48.479857 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:48.479968 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:48.479973 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:48.480646 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:48.480648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:48.481201 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:48.481208 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:48.481579 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:48.481581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:48.481638 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:48.482116 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:48.482127 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:48.484273 (+ 2146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":81,"threads_started":1}
W20260929 01:02:48.484553 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:48.479215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43805 (local address 127.0.98.62:36577)
0929 01:02:48.479376 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:48.479380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:48.479393 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:02:48.479507 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:48.479509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:48.479553 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:48.479620 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:48.480062 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:48.480558 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:48.481293 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:48.481490 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:48.481561 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:48.484430 (+ 2869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43805: BlockingRecv error: recv got EOF from 127.0.98.3:43805 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 01:02:49 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:49 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:49.090579 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:49.085200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.1:59777)
0929 01:02:49.085469 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:49.085482 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:49.085570 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:49.085816 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:49.085818 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:49.086001 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:49.086095 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:49.086100 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:49.086820 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:49.086823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:49.087379 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:49.087387 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:49.087831 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:49.087836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:49.087921 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:49.088476 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:49.088492 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:49.090429 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":95,"threads_started":1}
W20260929 01:02:49.090723 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:49.085288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59777 (local address 127.0.98.62:36577)
0929 01:02:49.085523 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:49.085526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:49.085539 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:02:49.085656 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:49.085658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:49.085701 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:49.085781 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:49.086190 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:49.086706 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:49.087478 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:49.087709 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:49.087783 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:49.090571 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59777: BlockingRecv error: recv got EOF from 127.0.98.1:59777 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":87,"threads_started":1}
Sep 29 01:02:49 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:49 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:49.299708 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:49.294052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.2:35851)
0929 01:02:49.294364 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:49.294386 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:49.294536 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:49.294834 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:49.294837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:49.295046 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:49.295137 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:49.295142 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:49.295899 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:49.295903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:49.296449 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:49.296457 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:49.296901 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:49.296904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:49.296985 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:49.297473 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:49.297485 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:49.299571 (+ 2086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":76,"threads_started":1}
W20260929 01:02:49.299815 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:49.294153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35851 (local address 127.0.98.62:36577)
0929 01:02:49.294321 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:49.294326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:49.294478 (+ 152us) server_negotiation.cc:407] Connection header received
0929 01:02:49.294637 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:49.294641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:49.294696 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:49.294786 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:49.295253 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:49.295780 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:49.296562 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:49.296784 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:49.296890 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:49.299693 (+ 2803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35851: BlockingRecv error: recv got EOF from 127.0.98.2:35851 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 29 01:02:49 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:49 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:49.490689 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:49.485362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.3:46205)
0929 01:02:49.485628 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:49.485641 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:49.485737 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:49.485965 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:49.485967 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:49.486141 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:49.486236 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:49.486241 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:49.487033 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:49.487036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:49.487584 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:49.487592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:49.487998 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:49.488001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:49.488057 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:49.488578 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:49.488589 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:49.490489 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":96,"threads_started":1}
W20260929 01:02:49.490760 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:49.485462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46205 (local address 127.0.98.62:36577)
0929 01:02:49.485597 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:49.485601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:49.485742 (+ 141us) server_negotiation.cc:407] Connection header received
0929 01:02:49.485815 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:49.485817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:49.485866 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:49.485929 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:49.486342 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:49.486943 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:49.487687 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:49.487905 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:49.487981 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:49.490641 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46205: BlockingRecv error: recv got EOF from 127.0.98.3:46205 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 01:02:50 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:50 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:50.100620 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:50.091597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.1:59423)
0929 01:02:50.091861 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:50.091877 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:50.092007 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:50.092356 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:50.092359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:50.092579 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:50.092702 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:50.092712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:50.093448 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:50.093450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:50.093995 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:50.094002 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:50.094384 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:50.094387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:50.094449 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:50.098720 (+ 4271us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:50.098732 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:50.100505 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":89,"threads_started":1}
W20260929 01:02:50.100771 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:50.091672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59423 (local address 127.0.98.62:36577)
0929 01:02:50.091947 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:50.091954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:50.091971 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:02:50.092102 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:50.092104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:50.092178 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:50.092285 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:50.092820 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:50.093352 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:50.094092 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:50.094296 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:50.094368 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:50.100639 (+ 6271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59423: BlockingRecv error: recv got EOF from 127.0.98.1:59423 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
Sep 29 01:02:50 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:50 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:50.305620 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:50.300513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.2:56297)
0929 01:02:50.300824 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:50.300840 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:50.300958 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:50.301214 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:50.301216 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:50.301400 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:50.301492 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:50.301497 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:50.302188 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:50.302191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:50.302732 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:50.302739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:50.303196 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:50.303199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:50.303259 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:50.303747 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:50.303757 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:50.305509 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":83,"threads_started":1}
W20260929 01:02:50.305792 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:50.300583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56297 (local address 127.0.98.62:36577)
0929 01:02:50.300702 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:50.300709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:50.300926 (+ 217us) server_negotiation.cc:407] Connection header received
0929 01:02:50.301057 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:50.301059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:50.301106 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:50.301175 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:50.301588 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:50.302100 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:50.302896 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:50.303102 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:50.303178 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:50.305639 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56297: BlockingRecv error: recv got EOF from 127.0.98.2:56297 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 29 01:02:50 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:50 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:50.496954 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:50.491671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.3:57685)
0929 01:02:50.491956 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:50.491969 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:50.492070 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:50.492313 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:50.492315 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:50.492497 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:50.492588 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:50.492594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:50.493289 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:50.493291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:50.493837 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:50.493844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:50.494236 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:50.494238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:50.494298 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:50.494774 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:50.494788 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:50.496782 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":115,"threads_started":1}
W20260929 01:02:50.496989 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:50.491759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57685 (local address 127.0.98.62:36577)
0929 01:02:50.491891 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:50.491895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:50.492043 (+ 148us) server_negotiation.cc:407] Connection header received
0929 01:02:50.492149 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:50.492151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:50.492200 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:50.492282 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:50.492682 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:50.493195 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:50.493936 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:50.494145 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:50.494222 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:50.496886 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57685: BlockingRecv error: recv got EOF from 127.0.98.3:57685 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 01:02:51 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:51 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:51.106568 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:51.101427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.1:36239)
0929 01:02:51.101728 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:51.101741 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:51.101836 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:51.102070 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:51.102072 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:51.102241 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:51.102356 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:51.102362 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:51.103116 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:51.103119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:51.103712 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:51.103721 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:51.104163 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:51.104166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:51.104230 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:51.104788 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:51.104800 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:51.106455 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":91,"threads_started":1}
W20260929 01:02:51.106710 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:51.101497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36239 (local address 127.0.98.62:36577)
0929 01:02:51.101729 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:51.101733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:51.101791 (+ 58us) server_negotiation.cc:407] Connection header received
0929 01:02:51.101917 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:51.101919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:51.101961 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:51.102036 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:51.102455 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:51.103004 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:51.103826 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:51.104059 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:51.104133 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:51.106580 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36239: BlockingRecv error: recv got EOF from 127.0.98.1:36239 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":85,"threads_started":1}
Sep 29 01:02:51 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:51 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:51.311933 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:51.306440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.2:59731)
0929 01:02:51.306755 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:51.306771 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:51.306930 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:51.307246 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:51.307249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:51.307477 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:51.307572 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:51.307577 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:51.308423 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:51.308425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:51.308996 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:51.309004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:51.309417 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:51.309419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:51.309479 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:51.309958 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:51.309968 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:51.311754 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":121,"threads_started":1}
W20260929 01:02:51.311969 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:51.306595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59731 (local address 127.0.98.62:36577)
0929 01:02:51.306766 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:51.306770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:51.306865 (+ 95us) server_negotiation.cc:407] Connection header received
0929 01:02:51.307030 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:51.307032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:51.307104 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:51.307197 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:51.307686 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:51.308324 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:51.309095 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:51.309298 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:51.309390 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:51.311871 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59731: BlockingRecv error: recv got EOF from 127.0.98.2:59731 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 01:02:51 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:51 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:51.503315 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:51.497940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.3:33523)
0929 01:02:51.498216 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:51.498230 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:51.498332 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:51.498576 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:51.498579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:51.498760 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:51.498911 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:51.498917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:51.499601 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:51.499603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:51.500190 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:51.500198 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:51.500588 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:51.500590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:51.500650 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:51.501120 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:51.501132 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:51.503202 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":99,"threads_started":1}
W20260929 01:02:51.503472 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:51.497988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33523 (local address 127.0.98.62:36577)
0929 01:02:51.498124 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:51.498129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:51.498303 (+ 174us) server_negotiation.cc:407] Connection header received
0929 01:02:51.498398 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:51.498400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:51.498445 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:51.498525 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:51.499010 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:51.499509 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:51.500297 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:51.500499 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:51.500570 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:51.503324 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33523: BlockingRecv error: recv got EOF from 127.0.98.3:33523 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 01:02:52 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:52 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:52.112821 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:52.107388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.1:42849)
0929 01:02:52.107654 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:52.107668 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:52.107754 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:52.108117 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:52.108120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:52.108329 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:52.108427 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:52.108433 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:52.109214 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:52.109218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:52.109775 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:52.109782 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:52.110239 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:52.110242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:52.110327 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:52.110875 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:52.110890 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:52.112654 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":70,"threads_started":1}
W20260929 01:02:52.112959 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:52.107466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42849 (local address 127.0.98.62:36577)
0929 01:02:52.107731 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:52.107735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:52.107748 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:02:52.107839 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:52.107841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:52.107891 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:52.108105 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:52.108548 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:52.109090 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:52.109902 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:52.110126 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:52.110237 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:52.112822 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42849: BlockingRecv error: recv got EOF from 127.0.98.1:42849 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":75,"threads_started":1}
Sep 29 01:02:52 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:52 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:52.317952 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:52.312662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.2:38843)
0929 01:02:52.312951 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:52.312964 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:52.313062 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:52.313339 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:52.313341 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:52.313514 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:52.313607 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:52.313612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:52.314307 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:52.314309 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:52.314941 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:52.314951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:52.315408 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:52.315412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:52.315509 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:52.316058 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:52.316069 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:52.317840 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":127,"threads_started":1}
W20260929 01:02:52.318140 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:52.312736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38843 (local address 127.0.98.62:36577)
0929 01:02:52.312920 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:52.312925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:52.313034 (+ 109us) server_negotiation.cc:407] Connection header received
0929 01:02:52.313170 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:52.313172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:52.313217 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:52.313285 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:52.313699 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:52.314220 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:52.315040 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:52.315253 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:52.315386 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:52.317985 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38843: BlockingRecv error: recv got EOF from 127.0.98.2:38843 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 01:02:52 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:52 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:52.510388 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:52.504210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.3:34657)
0929 01:02:52.504554 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:52.504566 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:52.504716 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:52.505026 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:52.505029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:52.505246 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:52.505340 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:52.505346 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:52.506170 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:52.506174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:52.506795 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:52.506877 (+ 82us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:52.507464 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:52.507468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:52.507587 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:52.508276 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:52.508288 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:52.510266 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":74,"threads_started":1}
W20260929 01:02:52.510601 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:52.504293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34657 (local address 127.0.98.62:36577)
0929 01:02:52.504456 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:52.504461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:52.504655 (+ 194us) server_negotiation.cc:407] Connection header received
0929 01:02:52.504826 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:52.504830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:52.504904 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:52.505003 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:52.505486 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:52.506038 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:52.507002 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:52.507306 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:52.507416 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:52.510470 (+ 3054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34657: BlockingRecv error: recv got EOF from 127.0.98.3:34657 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 01:02:53 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:53 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:53.119192 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:53.113581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.1:37903)
0929 01:02:53.113956 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:53.113969 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:53.114066 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:53.114298 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:53.114300 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:53.114476 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:53.114566 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:53.114571 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:53.115371 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:53.115374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:53.116012 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:53.116021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:53.116417 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:53.116419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:53.116473 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:53.117000 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:53.117011 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:53.119080 (+ 2069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":105,"threads_started":1}
W20260929 01:02:53.119382 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:53.113647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37903 (local address 127.0.98.62:36577)
0929 01:02:53.113892 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:53.113896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:53.114036 (+ 140us) server_negotiation.cc:407] Connection header received
0929 01:02:53.114132 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:53.114134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:53.114177 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:53.114255 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:53.114662 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:53.115278 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:53.116101 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:53.116320 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:53.116404 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:53.119211 (+ 2807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37903: BlockingRecv error: recv got EOF from 127.0.98.1:37903 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":83,"threads_started":1}
Sep 29 01:02:53 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:53 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:53.324546 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:53.318658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.2:33879)
0929 01:02:53.319042 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:53.319056 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:53.319275 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:53.319557 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:53.319560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:53.319777 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:53.319867 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:53.319873 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:53.320653 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:53.320657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:53.321239 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:53.321247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:53.321689 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:53.321692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:53.321769 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:53.322262 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:53.322274 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:53.324367 (+ 2093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":108,"threads_started":1}
W20260929 01:02:53.324591 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:53.318820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33879 (local address 127.0.98.62:36577)
0929 01:02:53.318996 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:53.319001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:53.319152 (+ 151us) server_negotiation.cc:407] Connection header received
0929 01:02:53.319382 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:53.319386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:53.319440 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:53.319540 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:53.319990 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:53.320538 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:53.321361 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:53.321580 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:53.321695 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:53.324471 (+ 2776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33879: BlockingRecv error: recv got EOF from 127.0.98.2:33879 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 01:02:53 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:53 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:53.516614 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:53.511124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.3:41141)
0929 01:02:53.511457 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:53.511470 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:53.511595 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:53.511867 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:53.511870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:53.512078 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:53.512166 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:53.512171 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:53.512906 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:53.512911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:53.513481 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:53.513489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:53.513934 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:53.513937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:53.514019 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:53.514600 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:53.514611 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:53.516504 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":84,"threads_started":1}
W20260929 01:02:53.516824 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:53.511212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41141 (local address 127.0.98.62:36577)
0929 01:02:53.511345 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:53.511350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:53.511545 (+ 195us) server_negotiation.cc:407] Connection header received
0929 01:02:53.511701 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:53.511704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:53.511758 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:53.511850 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:53.512281 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:53.512800 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:53.513599 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:53.513826 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:53.513933 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:53.516673 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41141: BlockingRecv error: recv got EOF from 127.0.98.3:41141 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 01:02:54 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:54 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:54.125675 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:54.119948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.1:40545)
0929 01:02:54.120325 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:54.120346 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:54.120436 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:54.120763 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:54.120768 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:54.121003 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:54.121116 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:54.121122 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:54.121883 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:54.121886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:54.122446 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:54.122454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:54.122918 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:54.122922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:54.122992 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:54.123627 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:54.123647 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:54.125533 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":95,"threads_started":1}
W20260929 01:02:54.125801 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:54.120096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40545 (local address 127.0.98.62:36577)
0929 01:02:54.120373 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:54.120378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:54.120395 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:02:54.120535 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:54.120538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:54.120593 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:54.120702 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:54.121222 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:54.121760 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:54.122548 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:54.122760 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:54.122895 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:54.125681 (+ 2786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40545: BlockingRecv error: recv got EOF from 127.0.98.1:40545 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":81,"threads_started":1}
Sep 29 01:02:54 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:54 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:54.331333 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:54.325450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.2:38567)
0929 01:02:54.325707 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:54.325742 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:54.325844 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:54.326248 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:54.326250 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:54.326424 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:54.326552 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:54.326559 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:54.327726 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:54.327729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:54.328313 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:54.328320 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:54.328743 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:54.328746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:54.328813 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:54.329296 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:54.329308 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:54.331157 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":101,"threads_started":1}
W20260929 01:02:54.331367 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:54.325495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38567 (local address 127.0.98.62:36577)
0929 01:02:54.325660 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:54.325665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:54.325824 (+ 159us) server_negotiation.cc:407] Connection header received
0929 01:02:54.325952 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:54.325955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:54.326012 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:54.326098 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:54.326675 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:54.327623 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:54.328412 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:54.328616 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:54.328689 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:54.331271 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38567: BlockingRecv error: recv got EOF from 127.0.98.2:38567 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 01:02:54 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:54 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:54.523180 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:54.517363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.3:51983)
0929 01:02:54.517635 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:54.517652 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:54.517757 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:54.518009 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:54.518011 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:54.518276 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:54.518383 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:54.518390 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:54.519208 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:54.519210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:54.519782 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:54.519790 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:54.520289 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:54.520294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:54.520396 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:54.521039 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:54.521052 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:54.523012 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":104,"threads_started":1}
W20260929 01:02:54.523214 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:54.517446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51983 (local address 127.0.98.62:36577)
0929 01:02:54.517581 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:54.517585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:54.517724 (+ 139us) server_negotiation.cc:407] Connection header received
0929 01:02:54.517852 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:54.517854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:54.517899 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:54.517969 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:54.518487 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:54.519078 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:54.519884 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:54.520145 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:54.520242 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:54.523120 (+ 2878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51983: BlockingRecv error: recv got EOF from 127.0.98.3:51983 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 01:02:55 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:55 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:55.138335 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:55.126482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.1:60265)
0929 01:02:55.126750 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:55.126764 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:55.126887 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:55.127144 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:55.127146 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:55.127326 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:55.127451 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:55.127457 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:55.128179 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:55.128183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:55.128916 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:55.128931 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:55.129611 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:55.129633 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:55.129736 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:55.136291 (+ 6555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:55.136304 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:55.138217 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":93,"threads_started":1}
W20260929 01:02:55.138540 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:55.126521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60265 (local address 127.0.98.62:36577)
0929 01:02:55.126770 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:55.126774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:55.126844 (+ 70us) server_negotiation.cc:407] Connection header received
0929 01:02:55.126954 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:55.126956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:55.126998 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:55.127076 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:55.127558 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:55.128083 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:55.129057 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:55.129467 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:55.129589 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:55.138364 (+ 8775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60265: BlockingRecv error: recv got EOF from 127.0.98.1:60265 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":106,"threads_started":1}
Sep 29 01:02:55 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:55 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:55.337435 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:55.332165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.2:54973)
0929 01:02:55.332473 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:55.332488 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:55.332602 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:55.332841 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:55.332843 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:55.333058 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:55.333246 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:55.333254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:55.333981 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:55.333984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:55.334559 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:55.334567 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:55.334998 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:55.335001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:55.335064 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:55.335531 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:55.335542 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:55.337323 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":92,"threads_started":1}
W20260929 01:02:55.337600 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:55.332347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54973 (local address 127.0.98.62:36577)
0929 01:02:55.332509 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:55.332513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:55.332567 (+ 54us) server_negotiation.cc:407] Connection header received
0929 01:02:55.332689 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:55.332691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:55.332733 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:55.332801 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:55.333354 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:55.333866 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:55.334659 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:55.334898 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:55.334977 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:55.337445 (+ 2468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54973: BlockingRecv error: recv got EOF from 127.0.98.2:54973 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 01:02:55 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:55 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:55.529536 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:55.524021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.3:51985)
0929 01:02:55.524352 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:55.524368 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:55.524490 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:55.524821 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:55.524825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:55.525056 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:55.525171 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:55.525177 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:55.525955 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:55.525961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:55.526530 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:55.526537 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:55.527059 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:55.527062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:55.527146 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:55.527621 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:55.527632 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:55.529370 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":101,"threads_started":1}
W20260929 01:02:55.529595 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:55.524096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51985 (local address 127.0.98.62:36577)
0929 01:02:55.524228 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:55.524232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:55.524429 (+ 197us) server_negotiation.cc:407] Connection header received
0929 01:02:55.524591 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:55.524594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:55.524674 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:55.524762 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:55.525321 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:55.525848 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:55.526645 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:55.526936 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:55.527054 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:55.529490 (+ 2436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51985: BlockingRecv error: recv got EOF from 127.0.98.3:51985 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 01:02:56 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:56 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:56.144382 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:56.139132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.1:49171)
0929 01:02:56.139417 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:56.139431 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:56.139511 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:56.139751 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:56.139753 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:56.139941 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:56.140033 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:56.140037 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.140724 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.140726 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:56.141277 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:56.141284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.141671 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.141674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:56.141729 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:56.142299 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:56.142311 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:56.144212 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":70,"threads_started":1}
W20260929 01:02:56.144438 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:56.139193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49171 (local address 127.0.98.62:36577)
0929 01:02:56.139471 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:56.139474 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:56.139487 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:02:56.139598 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:56.139600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:56.139643 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:56.139717 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:56.140129 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.140639 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.141373 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.141584 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.141655 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:56.144325 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49171: BlockingRecv error: recv got EOF from 127.0.98.1:49171 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":111,"threads_started":1}
Sep 29 01:02:56 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:56 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:56.344084 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:56.338317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.2:53577)
0929 01:02:56.338581 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:56.338595 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:56.338692 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:56.338949 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:56.338951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:56.339125 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:56.339216 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:56.339221 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.339919 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.339921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:56.340455 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:56.340462 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.340949 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.340953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:56.341037 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:56.341679 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:56.341690 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:56.343881 (+ 2191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
W20260929 01:02:56.344753 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:56.338377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53577 (local address 127.0.98.62:36577)
0929 01:02:56.338533 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:56.338540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:56.338675 (+ 135us) server_negotiation.cc:407] Connection header received
0929 01:02:56.338773 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:56.338775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:56.338844 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:56.338907 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:56.339314 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.339829 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.340588 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.340827 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.340936 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:56.344619 (+ 3683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53577: BlockingRecv error: recv got EOF from 127.0.98.2:53577 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 01:02:56 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:56 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:56.535745 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:56.530437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.3:56099)
0929 01:02:56.530707 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:56.530720 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:56.530831 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:56.531066 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:56.531067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:56.531242 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:56.531334 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:56.531339 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.532050 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.532052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:56.532606 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:56.532613 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.533018 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.533020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:56.533077 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:56.533635 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:56.533647 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:56.535576 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":99,"threads_started":1}
W20260929 01:02:56.535789 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:56.530589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56099 (local address 127.0.98.62:36577)
0929 01:02:56.530716 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:56.530720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:56.530768 (+ 48us) server_negotiation.cc:407] Connection header received
0929 01:02:56.530913 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:56.530915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:56.530959 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:56.531028 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:56.531441 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.531961 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.532708 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:56.532922 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:56.532998 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:56.535689 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56099: BlockingRecv error: recv got EOF from 127.0.98.3:56099 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 01:02:57 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:57 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:57.150712 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:57.145191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.1:59867)
0929 01:02:57.145475 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:57.145489 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:57.145587 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:57.146009 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:57.146011 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:57.146187 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:57.146315 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:57.146323 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:57.147366 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:57.147368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:57.147928 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:57.147936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:57.148321 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:57.148323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:57.148385 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:57.148848 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:57.148859 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:57.150580 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":108,"threads_started":1}
W20260929 01:02:57.150864 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:57.145235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59867 (local address 127.0.98.62:36577)
0929 01:02:57.145473 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:57.145476 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:57.145555 (+ 79us) server_negotiation.cc:407] Connection header received
0929 01:02:57.145689 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:57.145692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:57.145754 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:57.145881 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:57.146444 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:57.147268 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:57.148020 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:57.148229 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:57.148301 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:57.150707 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59867: BlockingRecv error: recv got EOF from 127.0.98.1:59867 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":101,"threads_started":1}
Sep 29 01:02:57 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:57 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:57.350350 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:57.345250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.2:48117)
0929 01:02:57.345516 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:57.345529 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:57.345622 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:57.345829 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:57.345831 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:57.346025 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:57.346159 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:57.346172 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:57.346912 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:57.346915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:57.347451 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:57.347459 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:57.347873 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:57.347876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:57.347937 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:57.348492 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:57.348504 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:57.350218 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":106,"threads_started":1}
W20260929 01:02:57.350430 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:57.345328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48117 (local address 127.0.98.62:36577)
0929 01:02:57.345476 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:57.345480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:57.345576 (+ 96us) server_negotiation.cc:407] Connection header received
0929 01:02:57.345685 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:57.345688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:57.345729 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:57.345810 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:57.346263 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:57.346765 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:57.347546 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:57.347751 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:57.347838 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:57.350339 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48117: BlockingRecv error: recv got EOF from 127.0.98.2:48117 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 01:02:57 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:57 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:57.541915 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:57.536507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.3:36115)
0929 01:02:57.536776 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:57.536790 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:57.536891 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:57.537100 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:57.537102 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:57.537335 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:57.537430 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:57.537436 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:57.538116 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:57.538118 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:57.538707 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:57.538714 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:57.539314 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:57.539317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:57.539387 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:57.539899 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:57.539910 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:57.541784 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":100,"threads_started":1}
W20260929 01:02:57.542002 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:57.536559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36115 (local address 127.0.98.62:36577)
0929 01:02:57.536702 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:57.536707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:57.536845 (+ 138us) server_negotiation.cc:407] Connection header received
0929 01:02:57.536957 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:57.536960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:57.537005 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:57.537069 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:57.537524 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:57.538032 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:57.538836 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:57.539115 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:57.539206 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:57.541914 (+ 2708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36115: BlockingRecv error: recv got EOF from 127.0.98.3:36115 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 01:02:58 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:58 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:58.156644 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:58.151592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.1:52017)
0929 01:02:58.151900 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:58.151914 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:58.152011 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:58.152251 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:58.152253 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:58.152429 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:58.152521 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:58.152526 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:58.153210 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:58.153212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:58.153755 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:58.153763 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:58.154147 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:58.154149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:58.154206 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:58.154693 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:58.154703 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:58.156531 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":70,"threads_started":1}
W20260929 01:02:58.156845 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:58.151648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52017 (local address 127.0.98.62:36577)
0929 01:02:58.151935 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:58.151938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:58.151970 (+ 32us) server_negotiation.cc:407] Connection header received
0929 01:02:58.152097 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:58.152099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:58.152142 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:58.152215 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:58.152613 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:58.153119 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:58.153851 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:58.154057 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:58.154128 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:58.156713 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52017: BlockingRecv error: recv got EOF from 127.0.98.1:52017 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":114,"threads_started":1}
Sep 29 01:02:58 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:58 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:58.356889 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:58.351320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.2:39303)
0929 01:02:58.351627 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:58.351640 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:58.351742 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:58.352027 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:58.352029 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:58.352299 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:58.352403 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:58.352409 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:58.353306 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:58.353312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:58.353916 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:58.353925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:58.354351 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:58.354355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:58.354434 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:58.355010 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:58.355021 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:58.356780 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":129,"threads_started":1}
W20260929 01:02:58.357069 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:58.351370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39303 (local address 127.0.98.62:36577)
0929 01:02:58.351565 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:58.351569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:58.351717 (+ 148us) server_negotiation.cc:407] Connection header received
0929 01:02:58.351834 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:58.351837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:58.351882 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:58.351950 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:58.352553 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:58.353203 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:58.354033 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:58.354247 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:58.354351 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:58.356912 (+ 2561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39303: BlockingRecv error: recv got EOF from 127.0.98.2:39303 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 01:02:58 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:58 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:58.548254 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:58.542744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.3:41293)
0929 01:02:58.543061 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:58.543075 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:58.543205 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:58.543482 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:58.543485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:58.543694 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:58.543784 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:58.543789 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:58.544555 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:58.544561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:58.545134 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:58.545141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:58.545593 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:58.545596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:58.545680 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:58.546170 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:58.546181 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:58.548113 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":79,"threads_started":1}
W20260929 01:02:58.548520 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:58.542835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41293 (local address 127.0.98.62:36577)
0929 01:02:58.542988 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:58.543015 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:58.543153 (+ 138us) server_negotiation.cc:407] Connection header received
0929 01:02:58.543311 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:58.543315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:58.543375 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:58.543461 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:58.543924 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:58.544450 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:58.545250 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:58.545483 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:58.545592 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:58.548258 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41293: BlockingRecv error: recv got EOF from 127.0.98.3:41293 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 01:02:59 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:59 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:59.162958 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:59.157538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.1:59811)
0929 01:02:59.157827 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:59.157841 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:59.157932 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:59.158173 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:59.158175 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:59.158360 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:59.158467 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:59.158472 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:59.159200 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:59.159202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:59.159752 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:59.159759 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:59.160184 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:59.160187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:59.160249 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:59.160820 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:59.160832 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:59.162753 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":83,"threads_started":1}
W20260929 01:02:59.163015 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:59.157637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59811 (local address 127.0.98.62:36577)
0929 01:02:59.157888 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:59.157891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:59.157905 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:02:59.158016 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:59.158018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:59.158062 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:59.158135 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:59.158563 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:59.159110 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:59.159847 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:59.160074 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:59.160145 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:59.162898 (+ 2753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59811: BlockingRecv error: recv got EOF from 127.0.98.1:59811 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
Sep 29 01:02:59 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:59 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:59.362979 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:59.357577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.2:45075)
0929 01:02:59.357847 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:59.357861 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:59.357945 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:59.358174 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:59.358176 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:59.358361 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:59.358453 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:59.358458 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:59.359331 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:59.359335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:59.360050 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:59.360059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:59.360555 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:59.360558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:59.360641 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:59.361133 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:59.361143 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:59.362777 (+ 1634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":102,"threads_started":1}
W20260929 01:02:59.363013 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:59.357756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45075 (local address 127.0.98.62:36577)
0929 01:02:59.357885 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:59.357888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:59.357903 (+ 15us) server_negotiation.cc:407] Connection header received
0929 01:02:59.358027 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:59.358029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:59.358070 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:59.358138 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:59.358545 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:59.359230 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:59.360159 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:59.360445 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:59.360552 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:59.362922 (+ 2370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45075: BlockingRecv error: recv got EOF from 127.0.98.2:45075 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 01:02:59 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:02:59 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:02:59.554948 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:02:59.549158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.3:52537)
0929 01:02:59.549435 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 01:02:59.549447 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:02:59.549575 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:02:59.549845 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:02:59.549848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:02:59.550082 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:02:59.550183 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:59.550190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:59.551096 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:59.551101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:59.551668 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:02:59.551676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:02:59.552128 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:02:59.552131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:02:59.552215 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:59.552764 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:02:59.552785 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:02:59.554740 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":80,"threads_started":1}
W20260929 01:02:59.555016 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:02:59.549212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52537 (local address 127.0.98.62:36577)
0929 01:02:59.549375 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 01:02:59.549380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:02:59.549515 (+ 135us) server_negotiation.cc:407] Connection header received
0929 01:02:59.549670 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:02:59.549675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:02:59.549731 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:02:59.549817 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:02:59.550298 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:59.550990 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:59.551787 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:02:59.552018 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:02:59.552126 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:02:59.554927 (+ 2801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52537: BlockingRecv error: recv got EOF from 127.0.98.3:52537 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 29 01:03:00 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:03:00 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:03:00.174542 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:00.163707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.1:35717)
0929 01:03:00.163983 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:00.163996 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:00.164106 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:00.164299 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:00.164301 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:00.164475 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:03:00.164566 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:00.164571 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:00.165277 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:00.165280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:00.165822 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:00.165829 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:00.166234 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:00.166237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:00.166296 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:00.172265 (+ 5969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:03:00.172308 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:03:00.174414 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":123,"threads_started":1}
W20260929 01:03:00.174688 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:03:00.163755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35717 (local address 127.0.98.62:36577)
0929 01:03:00.164100 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:00.164103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:00.164116 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:03:00.164171 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:00.164173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:00.164214 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:00.164291 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:00.164658 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:00.165187 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:00.165933 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:00.166143 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:00.166216 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:00.174554 (+ 8338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35717: BlockingRecv error: recv got EOF from 127.0.98.1:35717 (error 108)
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":91,"threads_started":1}
Sep 29 01:03:00 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:03:00 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:03:00.369460 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:00.363929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.2:37933)
0929 01:03:00.364250 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:00.364262 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:00.364366 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:00.364587 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:00.364589 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:00.364771 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:03:00.364863 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:00.364868 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:00.365562 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:00.365565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:00.366112 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:00.366119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:00.366523 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:00.366525 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:00.366592 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:00.367115 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:03:00.367173 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:03:00.369285 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":101,"threads_started":1}
W20260929 01:03:00.369572 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:03:00.363980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37933 (local address 127.0.98.62:36577)
0929 01:03:00.364128 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:00.364133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:00.364350 (+ 217us) server_negotiation.cc:407] Connection header received
0929 01:03:00.364445 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:00.364448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:00.364493 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:00.364550 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:00.364957 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:00.365470 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:00.366214 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:00.366427 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:00.366498 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:00.369457 (+ 2959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37933: BlockingRecv error: recv got EOF from 127.0.98.2:37933 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 01:03:00 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:03:00 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:03:00.560961 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:00.555730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.3:39375)
0929 01:03:00.556025 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:00.556039 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:00.556135 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:00.556408 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:00.556411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:00.556581 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:03:00.556674 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:00.556678 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:00.557406 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:00.557408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:00.557941 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:00.557948 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:00.558385 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:00.558387 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:00.558444 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:00.559119 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:03:00.559130 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:03:00.560839 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":135,"threads_started":1}
W20260929 01:03:00.561127 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:03:00.555789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39375 (local address 127.0.98.62:36577)
0929 01:03:00.555950 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:00.555955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:00.556117 (+ 162us) server_negotiation.cc:407] Connection header received
0929 01:03:00.556209 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:00.556212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:00.556279 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:00.556507 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:00.556779 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:00.557312 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:00.558060 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:00.558289 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:00.558366 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:00.560973 (+ 2607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39375: BlockingRecv error: recv got EOF from 127.0.98.3:39375 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 01:03:01 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:03:01 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:03:01.180864 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:01.175509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.1:56709)
0929 01:03:01.175781 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:01.175819 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:01.175921 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:01.176202 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:01.176204 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:01.176391 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:03:01.176485 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:01.176490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:01.177236 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:01.177238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:01.177788 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:01.177794 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:01.178210 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:01.178212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:01.178270 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:01.178753 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:03:01.178764 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:03:01.180694 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":99,"threads_started":1}
W20260929 01:03:01.180939 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:03:01.175593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56709 (local address 127.0.98.62:36577)
0929 01:03:01.175916 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:01.175919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:01.175932 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:03:01.176022 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:01.176027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:01.176085 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:01.176169 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:01.176592 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:01.177131 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:01.177893 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:01.178112 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:01.178204 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:01.180815 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56709: BlockingRecv error: recv got EOF from 127.0.98.1:56709 (error 108)
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":63,"threads_started":1}
Sep 29 01:03:01 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 01:03:01 dist-test-slave-7zdl krb5kdc[8543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 01:03:01.376124 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:01.370521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36577 (local address 127.0.98.2:35855)
0929 01:03:01.370788 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:01.370838 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:01.370931 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:01.371165 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:01.371167 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:01.371352 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:03:01.371443 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:01.371448 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:01.372253 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:01.372256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:01.372988 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:01.372999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:01.373495 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:01.373499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:01.373581 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:01.374062 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:03:01.374074 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:03:01.376012 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36577: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":98,"threads_started":1}
W20260929 01:03:01.376308 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:03:01.370651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35855 (local address 127.0.98.62:36577)
0929 01:03:01.370785 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:01.370788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:01.370907 (+ 119us) server_negotiation.cc:407] Connection header received
0929 01:03:01.371010 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:01.371012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:01.371055 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:01.371131 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:01.371550 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:01.372094 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:01.373086 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:01.373385 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:01.373494 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:01.376148 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35855: BlockingRecv error: recv got EOF from 127.0.98.2:35855 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 01:03:01.453518 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 9126
I20260929 01:03:01.459288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 9278
I20260929 01:03:01.464340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 9411
I20260929 01:03:01.469249 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 9056
2026-09-29T01:03:01Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17929 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T01:03:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:03:01Z Disabled control of system clock
I20260929 01:03:01.498749 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34617
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34617
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 01:03:01.606954 9606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:03:01.607214 9606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:03:01.607249 9606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:03:01.611207 9606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:03:01.611260 9606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:03:01.611276 9606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:03:01.611294 9606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:03:01.611310 9606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:03:01.616109 9606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34617
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:03:01.617151 9606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:03:01.618515 9606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:03:01.627619 9611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:01.627619 9612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:01.627866 9606 server_base.cc:1034] running on GCE node
W20260929 01:03:01.627964 9614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:01.628573 9606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:03:01.629748 9606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:03:01.630920 9606 hybrid_clock.cc:648] HybridClock initialized: now 1790643781630904 us; error 29 us; skew 500 ppm
I20260929 01:03:01.632789 9606 webserver.cc:460] Webserver started at http://127.0.98.62:44089/ using document root <none> and password file <none>
I20260929 01:03:01.633390 9606 fs_manager.cc:359] Metadata directory not provided
I20260929 01:03:01.633479 9606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:03:01.633692 9606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:03:01.635869 9606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "42a0b270e6dc4c56bdc3773b21f56fd2"
format_stamp: "Formatted at 2026-09-29 01:03:01 on dist-test-slave-7zdl"
server_key: "4adcdd369f550d94d965a9fd927e18a7"
server_key_iv: "e16d92e4b1420d96cdcacbfd39121a70"
server_key_version: "encryptionkey@0"
I20260929 01:03:01.636440 9606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "42a0b270e6dc4c56bdc3773b21f56fd2"
format_stamp: "Formatted at 2026-09-29 01:03:01 on dist-test-slave-7zdl"
server_key: "4adcdd369f550d94d965a9fd927e18a7"
server_key_iv: "e16d92e4b1420d96cdcacbfd39121a70"
server_key_version: "encryptionkey@0"
I20260929 01:03:01.640206 9606 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:03:01.642630 9620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:01.643949 9606 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:03:01.644155 9606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "42a0b270e6dc4c56bdc3773b21f56fd2"
format_stamp: "Formatted at 2026-09-29 01:03:01 on dist-test-slave-7zdl"
server_key: "4adcdd369f550d94d965a9fd927e18a7"
server_key_iv: "e16d92e4b1420d96cdcacbfd39121a70"
server_key_version: "encryptionkey@0"
I20260929 01:03:01.644286 9606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:03:01.668192 9606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:03:01.668980 9606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:03:01.669212 9606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:03:01.677023 9606 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34617
I20260929 01:03:01.677044 9672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34617 every 8 connection(s)
I20260929 01:03:01.678097 9606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:03:01.681209 9673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:03:01.684412 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 9606
I20260929 01:03:01.684512 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:03:01.684795 392 external_mini_cluster.cc:1428] Setting key 60f6f71cb57f27bef34f83d7b854328d
I20260929 01:03:01.687247 9673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42a0b270e6dc4c56bdc3773b21f56fd2: Bootstrap starting.
I20260929 01:03:01.689399 9673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42a0b270e6dc4c56bdc3773b21f56fd2: Neither blocks nor log segments found. Creating new log.
I20260929 01:03:01.690130 9673 log.cc:824] T 00000000000000000000000000000000 P 42a0b270e6dc4c56bdc3773b21f56fd2: Log is configured to *not* fsync() on all Append() calls
I20260929 01:03:01.692056 9673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42a0b270e6dc4c56bdc3773b21f56fd2: No bootstrap required, opened a new log
I20260929 01:03:01.694577 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:01.686318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43890 (local address 127.0.98.62:34617)
0929 01:03:01.687025 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:01.687041 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:01.687077 (+ 36us) server_negotiation.cc:407] Connection header received
0929 01:03:01.687767 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:01.687785 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:01.688176 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:01.688504 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:01.688926 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:01.689991 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:01.690920 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:01.691538 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:01.691714 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:01.692568 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:01.692574 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:01.692589 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:03:01.693065 (+ 476us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:01.693136 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:03:01.693160 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:01.693266 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:01.693399 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 01:03:01.693664 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":130,"threads_started":1}
I20260929 01:03:01.696395 9673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42a0b270e6dc4c56bdc3773b21f56fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a0b270e6dc4c56bdc3773b21f56fd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34617 } }
I20260929 01:03:01.696640 9673 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42a0b270e6dc4c56bdc3773b21f56fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:03:01.696691 9673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42a0b270e6dc4c56bdc3773b21f56fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42a0b270e6dc4c56bdc3773b21f56fd2, State: Initialized, Role: FOLLOWER
I20260929 01:03:01.697155 9673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42a0b270e6dc4c56bdc3773b21f56fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a0b270e6dc4c56bdc3773b21f56fd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34617 } }
I20260929 01:03:01.697307 9673 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42a0b270e6dc4c56bdc3773b21f56fd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:03:01.697371 9673 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42a0b270e6dc4c56bdc3773b21f56fd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:03:01.697485 9673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42a0b270e6dc4c56bdc3773b21f56fd2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:03:01.698333 9673 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42a0b270e6dc4c56bdc3773b21f56fd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a0b270e6dc4c56bdc3773b21f56fd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34617 } }
I20260929 01:03:01.698653 9673 leader_election.cc:302] T 00000000000000000000000000000000 P 42a0b270e6dc4c56bdc3773b21f56fd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42a0b270e6dc4c56bdc3773b21f56fd2; no voters:
I20260929 01:03:01.698957 9673 leader_election.cc:288] T 00000000000000000000000000000000 P 42a0b270e6dc4c56bdc3773b21f56fd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:03:01.699064 9678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42a0b270e6dc4c56bdc3773b21f56fd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:03:01.699360 9678 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42a0b270e6dc4c56bdc3773b21f56fd2 [term 1 LEADER]: Becoming Leader. State: Replica: 42a0b270e6dc4c56bdc3773b21f56fd2, State: Running, Role: LEADER
I20260929 01:03:01.699741 9678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42a0b270e6dc4c56bdc3773b21f56fd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a0b270e6dc4c56bdc3773b21f56fd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34617 } }
I20260929 01:03:01.700181 9673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42a0b270e6dc4c56bdc3773b21f56fd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:03:01.701473 9680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42a0b270e6dc4c56bdc3773b21f56fd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42a0b270e6dc4c56bdc3773b21f56fd2. Latest consensus state: current_term: 1 leader_uuid: "42a0b270e6dc4c56bdc3773b21f56fd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a0b270e6dc4c56bdc3773b21f56fd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34617 } } }
I20260929 01:03:01.701512 9679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42a0b270e6dc4c56bdc3773b21f56fd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42a0b270e6dc4c56bdc3773b21f56fd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42a0b270e6dc4c56bdc3773b21f56fd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34617 } } }
I20260929 01:03:01.701684 9680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42a0b270e6dc4c56bdc3773b21f56fd2 [sys.catalog]: This master's current role is: LEADER
I20260929 01:03:01.701722 9679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42a0b270e6dc4c56bdc3773b21f56fd2 [sys.catalog]: This master's current role is: LEADER
I20260929 01:03:01.702081 9689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:03:01.705454 9689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:03:01.713548 9689 catalog_manager.cc:1347] Generated new cluster ID: 9adbb45d4ea34e58a49e77cffea4515a
I20260929 01:03:01.713624 9689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:03:01.724534 9689 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:03:01.725734 9689 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:03:01.734247 9689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42a0b270e6dc4c56bdc3773b21f56fd2: Generated new TSK 0
I20260929 01:03:01.735000 9689 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:03:01.740111 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34617
--builtin_ntp_servers=127.0.98.20:38821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 01:03:01.854193 9697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:03:01.854449 9697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:03:01.854529 9697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:03:01.858632 9697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:03:01.858683 9697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:03:01.858770 9697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:03:01.863404 9697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:03:01.864449 9697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:03:01.865794 9697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:03:01.875177 9703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:01.875249 9702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:01.875303 9697 server_base.cc:1034] running on GCE node
W20260929 01:03:01.875202 9705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:01.875756 9697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:03:01.876442 9697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:03:01.877601 9697 hybrid_clock.cc:648] HybridClock initialized: now 1790643781877586 us; error 25 us; skew 500 ppm
I20260929 01:03:01.879433 9697 webserver.cc:460] Webserver started at http://127.0.98.1:41439/ using document root <none> and password file <none>
I20260929 01:03:01.880002 9697 fs_manager.cc:359] Metadata directory not provided
I20260929 01:03:01.880062 9697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:03:01.880294 9697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:03:01.882362 9697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "f7e219f8f07e46c1b2885a040a5b547e"
format_stamp: "Formatted at 2026-09-29 01:03:01 on dist-test-slave-7zdl"
server_key: "e3428a12e1cab90e209abb0fcabd49a8"
server_key_iv: "484beef60addd6f978c7fd11c72fbc8d"
server_key_version: "encryptionkey@0"
I20260929 01:03:01.882985 9697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f7e219f8f07e46c1b2885a040a5b547e"
format_stamp: "Formatted at 2026-09-29 01:03:01 on dist-test-slave-7zdl"
server_key: "e3428a12e1cab90e209abb0fcabd49a8"
server_key_iv: "484beef60addd6f978c7fd11c72fbc8d"
server_key_version: "encryptionkey@0"
I20260929 01:03:01.886653 9697 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:03:01.889168 9711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:01.890422 9697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:03:01.890560 9697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "f7e219f8f07e46c1b2885a040a5b547e"
format_stamp: "Formatted at 2026-09-29 01:03:01 on dist-test-slave-7zdl"
server_key: "e3428a12e1cab90e209abb0fcabd49a8"
server_key_iv: "484beef60addd6f978c7fd11c72fbc8d"
server_key_version: "encryptionkey@0"
I20260929 01:03:01.890681 9697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:03:01.907704 9697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:03:01.908416 9697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:03:01.908613 9697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:03:01.909209 9697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:03:01.910156 9697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:03:01.910228 9697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:01.910302 9697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:03:01.910347 9697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:01.918771 9697 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44049
I20260929 01:03:01.918854 9824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44049 every 8 connection(s)
I20260929 01:03:01.919734 9697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:03:01.926940 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 9697
I20260929 01:03:01.927052 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:03:01.927384 392 external_mini_cluster.cc:1428] Setting key c968a038cbe093240ab09125e0976382
I20260929 01:03:01.928905 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:01.921643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59457 (local address 127.0.98.62:34617)
0929 01:03:01.921804 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:01.921808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:01.922875 (+ 1067us) server_negotiation.cc:407] Connection header received
0929 01:03:01.924232 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:01.924235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:01.924279 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:01.924354 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:01.925166 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:01.925699 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:01.926758 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:01.927142 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:01.927260 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:01.928417 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:01.928419 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:01.928421 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:03:01.928433 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:01.928451 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:03:01.928458 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:01.928518 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:01.928709 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 01:03:01.928772 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 01:03:01.929366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34617
--builtin_ntp_servers=127.0.98.20:38821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 01:03:01.929821 9827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:01.922073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34617 (local address 127.0.98.1:59457)
0929 01:03:01.922703 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:01.922741 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:01.924055 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:01.924503 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:01.924510 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:01.924588 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:03:01.925052 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:01.925063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:01.925802 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:01.925805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:01.926640 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:01.926648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:01.927292 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:01.927295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:01.927457 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:01.928143 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:03:01.928145 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:03:01.928197 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:03:01.928279 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:03:01.928544 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:03:01.928553 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:03:01.928559 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 01:03:01.928706 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 01:03:01.928967 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":117,"threads_started":1}
I20260929 01:03:01.931083 9825 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34617
I20260929 01:03:01.931336 9825 heartbeater.cc:464] Registering TS with master...
I20260929 01:03:01.931921 9825 heartbeater.cc:511] Master 127.0.98.62:34617 requested a full tablet report, sending...
I20260929 01:03:01.933480 9637 ts_manager.cc:194] Registered new tserver with Master: f7e219f8f07e46c1b2885a040a5b547e (127.0.98.1:44049)
I20260929 01:03:01.935259 9637 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:59457
W20260929 01:03:02.044713 9828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:03:02.045038 9828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:03:02.045077 9828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:03:02.049221 9828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:03:02.049278 9828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:03:02.049335 9828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:03:02.053896 9828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:03:02.054864 9828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:03:02.056282 9828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:03:02.065563 9828 server_base.cc:1034] running on GCE node
W20260929 01:03:02.065481 9833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:02.065481 9834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:02.065512 9836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:02.066108 9828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:03:02.066727 9828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:03:02.067919 9828 hybrid_clock.cc:648] HybridClock initialized: now 1790643782067902 us; error 27 us; skew 500 ppm
I20260929 01:03:02.069708 9828 webserver.cc:460] Webserver started at http://127.0.98.2:35047/ using document root <none> and password file <none>
I20260929 01:03:02.070284 9828 fs_manager.cc:359] Metadata directory not provided
I20260929 01:03:02.070353 9828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:03:02.070607 9828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:03:02.072865 9828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "6e02ef4820c742548f9f0eec95bc5060"
format_stamp: "Formatted at 2026-09-29 01:03:02 on dist-test-slave-7zdl"
server_key: "17f2b0f0e0ad6d90523bd91cb9546b0e"
server_key_iv: "35b2a2460124cbb44405172d231f7be1"
server_key_version: "encryptionkey@0"
I20260929 01:03:02.073418 9828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6e02ef4820c742548f9f0eec95bc5060"
format_stamp: "Formatted at 2026-09-29 01:03:02 on dist-test-slave-7zdl"
server_key: "17f2b0f0e0ad6d90523bd91cb9546b0e"
server_key_iv: "35b2a2460124cbb44405172d231f7be1"
server_key_version: "encryptionkey@0"
I20260929 01:03:02.076961 9828 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:03:02.079838 9842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:02.081049 9828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:03:02.081182 9828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "6e02ef4820c742548f9f0eec95bc5060"
format_stamp: "Formatted at 2026-09-29 01:03:02 on dist-test-slave-7zdl"
server_key: "17f2b0f0e0ad6d90523bd91cb9546b0e"
server_key_iv: "35b2a2460124cbb44405172d231f7be1"
server_key_version: "encryptionkey@0"
I20260929 01:03:02.081296 9828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:03:02.096843 9828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:03:02.097502 9828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:03:02.097692 9828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:03:02.098243 9828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:03:02.099226 9828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:03:02.099296 9828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:02.099395 9828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:03:02.099448 9828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:02.108492 9828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35225
I20260929 01:03:02.108512 9955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35225 every 8 connection(s)
I20260929 01:03:02.109377 9828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:03:02.116098 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 9828
I20260929 01:03:02.116214 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:03:02.116503 392 external_mini_cluster.cc:1428] Setting key 3dd89adaca8747ba7811f336937e4124
I20260929 01:03:02.118625 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34617
--builtin_ntp_servers=127.0.98.20:38821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 01:03:02.119410 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:02.111209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45433 (local address 127.0.98.62:34617)
0929 01:03:02.111377 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:02.111382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:02.112401 (+ 1019us) server_negotiation.cc:407] Connection header received
0929 01:03:02.114229 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:02.114232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:02.114279 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:02.114360 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:02.115253 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:02.115774 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:02.117334 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:02.117630 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:02.117753 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:02.118944 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:02.118947 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:02.118949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:03:02.118961 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:02.118979 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:03:02.118987 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:02.119049 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:02.119255 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 01:03:02.119300 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:03:02.120477 9958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:02.111640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34617 (local address 127.0.98.2:45433)
0929 01:03:02.112269 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:02.112307 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:02.114038 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:02.114508 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:02.114515 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:02.114592 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:03:02.115135 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:02.115149 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:02.115905 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:02.115910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:02.117204 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:02.117214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:02.117781 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:02.117787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:02.117974 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:02.118585 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:03:02.118587 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:03:02.118640 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:03:02.118723 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:03:02.119074 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:03:02.119085 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:03:02.119094 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 01:03:02.119222 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 01:03:02.119517 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":112,"threads_started":1}
I20260929 01:03:02.121559 9956 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34617
I20260929 01:03:02.121790 9956 heartbeater.cc:464] Registering TS with master...
I20260929 01:03:02.122331 9956 heartbeater.cc:511] Master 127.0.98.62:34617 requested a full tablet report, sending...
I20260929 01:03:02.123347 9637 ts_manager.cc:194] Registered new tserver with Master: 6e02ef4820c742548f9f0eec95bc5060 (127.0.98.2:35225)
I20260929 01:03:02.123917 9637 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:45433
W20260929 01:03:02.229050 9959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:03:02.229324 9959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:03:02.229393 9959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:03:02.233443 9959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:03:02.233494 9959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:03:02.233583 9959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:03:02.238329 9959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:03:02.239383 9959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:03:02.240679 9959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:03:02.249975 9965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:02.249989 9967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:02.249996 9964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:02.250072 9959 server_base.cc:1034] running on GCE node
I20260929 01:03:02.250541 9959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:03:02.251199 9959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:03:02.252394 9959 hybrid_clock.cc:648] HybridClock initialized: now 1790643782252381 us; error 27 us; skew 500 ppm
I20260929 01:03:02.254060 9959 webserver.cc:460] Webserver started at http://127.0.98.3:39881/ using document root <none> and password file <none>
I20260929 01:03:02.254631 9959 fs_manager.cc:359] Metadata directory not provided
I20260929 01:03:02.254719 9959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:03:02.254964 9959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:03:02.256981 9959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "5876fdf45cf74f519b773b916fff60e7"
format_stamp: "Formatted at 2026-09-29 01:03:02 on dist-test-slave-7zdl"
server_key: "3f0044f8d041c5d24489374cbf2d4e34"
server_key_iv: "dc910cf9c63d8be882a663bfece4aecd"
server_key_version: "encryptionkey@0"
I20260929 01:03:02.257516 9959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5876fdf45cf74f519b773b916fff60e7"
format_stamp: "Formatted at 2026-09-29 01:03:02 on dist-test-slave-7zdl"
server_key: "3f0044f8d041c5d24489374cbf2d4e34"
server_key_iv: "dc910cf9c63d8be882a663bfece4aecd"
server_key_version: "encryptionkey@0"
I20260929 01:03:02.261067 9959 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 01:03:02.263515 9973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:02.264698 9959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 01:03:02.264829 9959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "5876fdf45cf74f519b773b916fff60e7"
format_stamp: "Formatted at 2026-09-29 01:03:02 on dist-test-slave-7zdl"
server_key: "3f0044f8d041c5d24489374cbf2d4e34"
server_key_iv: "dc910cf9c63d8be882a663bfece4aecd"
server_key_version: "encryptionkey@0"
I20260929 01:03:02.264937 9959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:03:02.284996 9959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:03:02.285712 9959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:03:02.285936 9959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:03:02.286509 9959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:03:02.287460 9959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:03:02.287531 9959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:02.287603 9959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:03:02.287659 9959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:02.296501 9959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34343
I20260929 01:03:02.296530 10086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34343 every 8 connection(s)
I20260929 01:03:02.297372 9959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:03:02.305119 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 9959
I20260929 01:03:02.305243 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:03:02.305543 392 external_mini_cluster.cc:1428] Setting key 152a6ed2fa6beff86ea31d669507641e
I20260929 01:03:02.306396 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:02.299103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37023 (local address 127.0.98.62:34617)
0929 01:03:02.299226 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:02.299230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:02.300227 (+ 997us) server_negotiation.cc:407] Connection header received
0929 01:03:02.301734 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:02.301738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:02.301797 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:02.301878 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:02.302718 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:02.303272 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:02.304406 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:02.304634 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:02.304745 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:02.305901 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:02.305905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:02.305909 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:03:02.305927 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:02.305944 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:03:02.305952 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:02.306037 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:02.306176 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 01:03:02.306237 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 01:03:02.307451 10089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:02.299443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34617 (local address 127.0.98.3:37023)
0929 01:03:02.300086 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:02.300125 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:02.301533 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:02.302042 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:02.302050 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:02.302127 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:03:02.302583 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:02.302593 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:02.303408 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:02.303418 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:02.304284 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:02.304293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:02.304782 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:02.304785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:02.304948 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:02.305604 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:03:02.305607 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:03:02.305666 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:03:02.305735 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:03:02.306041 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:03:02.306048 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:03:02.306054 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 01:03:02.306178 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 01:03:02.306496 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":106,"threads_started":1}
I20260929 01:03:02.308442 10087 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34617
I20260929 01:03:02.308650 10087 heartbeater.cc:464] Registering TS with master...
I20260929 01:03:02.309132 10087 heartbeater.cc:511] Master 127.0.98.62:34617 requested a full tablet report, sending...
I20260929 01:03:02.310058 9637 ts_manager.cc:194] Registered new tserver with Master: 5876fdf45cf74f519b773b916fff60e7 (127.0.98.3:34343)
I20260929 01:03:02.310554 9637 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:37023
I20260929 01:03:02.319334 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 01:03:02.321950 9676 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:03:02.321150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43906 (local address 127.0.98.62:34617)
0929 01:03:02.321273 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:02.321277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:02.321424 (+ 147us) server_negotiation.cc:407] Connection header received
0929 01:03:02.321507 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:02.321509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:02.321547 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:02.321612 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:02.321797 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43906: BlockingRecv error: recv got EOF from 127.0.0.1:43906 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 01:03:02.322124 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 9697
I20260929 01:03:02.328161 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 9828
I20260929 01:03:02.333552 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 9959
I20260929 01:03:02.338559 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 9606
2026-09-29T01:03:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (868 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T01:03:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:03:02Z Disabled control of system clock
I20260929 01:03:02.366634 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33777
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 01:03:02.474906 10098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:03:02.475167 10098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:03:02.475211 10098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:03:02.479151 10098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:03:02.479206 10098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:03:02.479221 10098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:03:02.479238 10098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:03:02.479254 10098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:03:02.484004 10098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33777
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:03:02.485031 10098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:03:02.486349 10098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:03:02.494639 10103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:02.494679 10106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:02.494640 10104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:02.494808 10098 server_base.cc:1034] running on GCE node
I20260929 01:03:02.495555 10098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:03:02.496618 10098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:03:02.497776 10098 hybrid_clock.cc:648] HybridClock initialized: now 1790643782497741 us; error 47 us; skew 500 ppm
I20260929 01:03:02.499536 10098 webserver.cc:460] Webserver started at http://127.0.98.62:33713/ using document root <none> and password file <none>
I20260929 01:03:02.500312 10098 fs_manager.cc:359] Metadata directory not provided
I20260929 01:03:02.500381 10098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:03:02.500597 10098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:03:02.502578 10098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "f13974a289c74cadab33045718b1eb6c"
format_stamp: "Formatted at 2026-09-29 01:03:02 on dist-test-slave-7zdl"
server_key: "b09fce84fc80d11c00d153c2399f3297"
server_key_iv: "e63afbf2fb7a1221cf1f0e21fd400d78"
server_key_version: "encryptionkey@0"
I20260929 01:03:02.503157 10098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "f13974a289c74cadab33045718b1eb6c"
format_stamp: "Formatted at 2026-09-29 01:03:02 on dist-test-slave-7zdl"
server_key: "b09fce84fc80d11c00d153c2399f3297"
server_key_iv: "e63afbf2fb7a1221cf1f0e21fd400d78"
server_key_version: "encryptionkey@0"
I20260929 01:03:02.506697 10098 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:03:02.509002 10112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:02.510162 10098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:03:02.510291 10098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "f13974a289c74cadab33045718b1eb6c"
format_stamp: "Formatted at 2026-09-29 01:03:02 on dist-test-slave-7zdl"
server_key: "b09fce84fc80d11c00d153c2399f3297"
server_key_iv: "e63afbf2fb7a1221cf1f0e21fd400d78"
server_key_version: "encryptionkey@0"
I20260929 01:03:02.510392 10098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:03:02.514722 10098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:03:02.515305 10098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:03:02.515475 10098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:03:02.522403 10098 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33777
I20260929 01:03:02.522421 10164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33777 every 8 connection(s)
I20260929 01:03:02.523360 10098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:03:02.526381 10165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:03:02.531495 10165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f13974a289c74cadab33045718b1eb6c: Bootstrap starting.
I20260929 01:03:02.531860 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 10098
I20260929 01:03:02.531944 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:03:02.532203 392 external_mini_cluster.cc:1428] Setting key 9ab5e4aed6aafb362afb79e813b518bd
I20260929 01:03:02.533981 10165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f13974a289c74cadab33045718b1eb6c: Neither blocks nor log segments found. Creating new log.
I20260929 01:03:02.534693 10165 log.cc:824] T 00000000000000000000000000000000 P f13974a289c74cadab33045718b1eb6c: Log is configured to *not* fsync() on all Append() calls
I20260929 01:03:02.536907 10165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f13974a289c74cadab33045718b1eb6c: No bootstrap required, opened a new log
I20260929 01:03:02.537664 10168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:02.533506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47804 (local address 127.0.98.62:33777)
0929 01:03:02.534122 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:02.534148 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:02.534182 (+ 34us) server_negotiation.cc:407] Connection header received
0929 01:03:02.534770 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:02.534786 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:02.535199 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:02.535506 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:02.536226 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:02.536232 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:02.536246 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:03:02.536266 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:02.536333 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:03:02.536355 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:02.536462 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:02.536558 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 01:03:02.536789 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":118,"threads_started":1}
I20260929 01:03:02.540879 10165 raft_consensus.cc:348] T 00000000000000000000000000000000 P f13974a289c74cadab33045718b1eb6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13974a289c74cadab33045718b1eb6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33777 } }
I20260929 01:03:02.541079 10165 raft_consensus.cc:374] T 00000000000000000000000000000000 P f13974a289c74cadab33045718b1eb6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:03:02.541118 10165 raft_consensus.cc:729] T 00000000000000000000000000000000 P f13974a289c74cadab33045718b1eb6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f13974a289c74cadab33045718b1eb6c, State: Initialized, Role: FOLLOWER
I20260929 01:03:02.541507 10165 consensus_queue.cc:260] T 00000000000000000000000000000000 P f13974a289c74cadab33045718b1eb6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13974a289c74cadab33045718b1eb6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33777 } }
I20260929 01:03:02.541626 10165 raft_consensus.cc:388] T 00000000000000000000000000000000 P f13974a289c74cadab33045718b1eb6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:03:02.541666 10165 raft_consensus.cc:482] T 00000000000000000000000000000000 P f13974a289c74cadab33045718b1eb6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:03:02.541728 10165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f13974a289c74cadab33045718b1eb6c [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:03:02.542672 10165 raft_consensus.cc:504] T 00000000000000000000000000000000 P f13974a289c74cadab33045718b1eb6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13974a289c74cadab33045718b1eb6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33777 } }
I20260929 01:03:02.543067 10165 leader_election.cc:302] T 00000000000000000000000000000000 P f13974a289c74cadab33045718b1eb6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f13974a289c74cadab33045718b1eb6c; no voters:
I20260929 01:03:02.543365 10165 leader_election.cc:288] T 00000000000000000000000000000000 P f13974a289c74cadab33045718b1eb6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:03:02.543570 10170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f13974a289c74cadab33045718b1eb6c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:03:02.543896 10170 raft_consensus.cc:686] T 00000000000000000000000000000000 P f13974a289c74cadab33045718b1eb6c [term 1 LEADER]: Becoming Leader. State: Replica: f13974a289c74cadab33045718b1eb6c, State: Running, Role: LEADER
I20260929 01:03:02.544181 10170 consensus_queue.cc:237] T 00000000000000000000000000000000 P f13974a289c74cadab33045718b1eb6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13974a289c74cadab33045718b1eb6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33777 } }
I20260929 01:03:02.544572 10165 sys_catalog.cc:564] T 00000000000000000000000000000000 P f13974a289c74cadab33045718b1eb6c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:03:02.545982 10171 sys_catalog.cc:455] T 00000000000000000000000000000000 P f13974a289c74cadab33045718b1eb6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f13974a289c74cadab33045718b1eb6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13974a289c74cadab33045718b1eb6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33777 } } }
I20260929 01:03:02.546008 10172 sys_catalog.cc:455] T 00000000000000000000000000000000 P f13974a289c74cadab33045718b1eb6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f13974a289c74cadab33045718b1eb6c. Latest consensus state: current_term: 1 leader_uuid: "f13974a289c74cadab33045718b1eb6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13974a289c74cadab33045718b1eb6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33777 } } }
I20260929 01:03:02.546113 10171 sys_catalog.cc:458] T 00000000000000000000000000000000 P f13974a289c74cadab33045718b1eb6c [sys.catalog]: This master's current role is: LEADER
I20260929 01:03:02.546115 10172 sys_catalog.cc:458] T 00000000000000000000000000000000 P f13974a289c74cadab33045718b1eb6c [sys.catalog]: This master's current role is: LEADER
I20260929 01:03:02.546491 10179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:03:02.549486 10179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:03:02.554703 10179 catalog_manager.cc:1347] Generated new cluster ID: 543fc3014a504bd2b60e27da8e99875b
I20260929 01:03:02.554778 10179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:03:02.566551 10179 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:03:02.566947 10179 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:03:02.579734 10179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f13974a289c74cadab33045718b1eb6c: Generated new TSK 0
I20260929 01:03:02.580402 10179 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:03:02.588891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33777
--builtin_ntp_servers=127.0.98.20:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 01:03:02.699273 10189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:03:02.699525 10189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:03:02.699564 10189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:03:02.703639 10189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:03:02.703696 10189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:03:02.703784 10189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:03:02.708676 10189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:03:02.709797 10189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:03:02.711272 10189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:03:02.720738 10197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:02.720743 10195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:02.720748 10194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:02.720918 10189 server_base.cc:1034] running on GCE node
I20260929 01:03:02.721460 10189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:03:02.722030 10189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:03:02.723201 10189 hybrid_clock.cc:648] HybridClock initialized: now 1790643782723185 us; error 27 us; skew 500 ppm
I20260929 01:03:02.725127 10189 webserver.cc:460] Webserver started at http://127.0.98.1:42119/ using document root <none> and password file <none>
I20260929 01:03:02.725665 10189 fs_manager.cc:359] Metadata directory not provided
I20260929 01:03:02.725744 10189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:03:02.725992 10189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:03:02.728015 10189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "d30f1b40cb2247e9b44913a8f39045d4"
format_stamp: "Formatted at 2026-09-29 01:03:02 on dist-test-slave-7zdl"
server_key: "9edc427b756f3d78e67ab04ba753a1c4"
server_key_iv: "64dffc8f93e0d3f77331000464e931f9"
server_key_version: "encryptionkey@0"
I20260929 01:03:02.728544 10189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d30f1b40cb2247e9b44913a8f39045d4"
format_stamp: "Formatted at 2026-09-29 01:03:02 on dist-test-slave-7zdl"
server_key: "9edc427b756f3d78e67ab04ba753a1c4"
server_key_iv: "64dffc8f93e0d3f77331000464e931f9"
server_key_version: "encryptionkey@0"
I20260929 01:03:02.731992 10189 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:03:02.734359 10203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:02.735636 10189 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:03:02.735774 10189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "d30f1b40cb2247e9b44913a8f39045d4"
format_stamp: "Formatted at 2026-09-29 01:03:02 on dist-test-slave-7zdl"
server_key: "9edc427b756f3d78e67ab04ba753a1c4"
server_key_iv: "64dffc8f93e0d3f77331000464e931f9"
server_key_version: "encryptionkey@0"
I20260929 01:03:02.735880 10189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:03:02.740262 10189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:03:02.740835 10189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:03:02.741011 10189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:03:02.741534 10189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:03:02.742401 10189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:03:02.742471 10189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:02.742537 10189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:03:02.742583 10189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:02.751492 10189 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38847
I20260929 01:03:02.751528 10316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38847 every 8 connection(s)
I20260929 01:03:02.752341 10189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:03:02.754909 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 10189
I20260929 01:03:02.755021 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:03:02.755328 392 external_mini_cluster.cc:1428] Setting key b4f668515f451752cc509a618d798bee
I20260929 01:03:02.757391 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33777
--builtin_ntp_servers=127.0.98.20:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 01:03:02.759397 10168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:02.753997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44253 (local address 127.0.98.62:33777)
0929 01:03:02.754137 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:02.754142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:02.755353 (+ 1211us) server_negotiation.cc:407] Connection header received
0929 01:03:02.756830 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:02.756833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:02.756873 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:02.756959 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:02.758223 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:02.758229 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:02.758252 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:03:02.758266 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:02.758292 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:03:02.758306 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:02.758397 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:02.758598 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 01:03:02.758674 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 01:03:02.759819 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:02.754470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33777 (local address 127.0.98.1:44253)
0929 01:03:02.755186 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:02.755238 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:02.756642 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:02.757126 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:02.757136 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:02.757247 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:03:02.757935 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:03:02.757937 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:03:02.757988 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:03:02.758067 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:03:02.758400 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:03:02.758408 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:03:02.758414 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 01:03:02.758612 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 01:03:02.758892 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":142,"threads_started":1}
I20260929 01:03:02.761010 10317 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33777
I20260929 01:03:02.761256 10317 heartbeater.cc:464] Registering TS with master...
I20260929 01:03:02.761696 10317 heartbeater.cc:511] Master 127.0.98.62:33777 requested a full tablet report, sending...
I20260929 01:03:02.763193 10129 ts_manager.cc:194] Registered new tserver with Master: d30f1b40cb2247e9b44913a8f39045d4 (127.0.98.1:38847)
W20260929 01:03:02.871610 10320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:03:02.871876 10320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:03:02.871914 10320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:03:02.875931 10320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:03:02.875981 10320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:03:02.876061 10320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:03:02.880755 10320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:03:02.881821 10320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:03:02.883258 10320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:03:02.893061 10325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:02.893062 10326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:02.893136 10328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:02.893303 10320 server_base.cc:1034] running on GCE node
I20260929 01:03:02.893828 10320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:03:02.894459 10320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:03:02.895648 10320 hybrid_clock.cc:648] HybridClock initialized: now 1790643782895597 us; error 72 us; skew 500 ppm
I20260929 01:03:02.897419 10320 webserver.cc:460] Webserver started at http://127.0.98.2:33099/ using document root <none> and password file <none>
I20260929 01:03:02.898015 10320 fs_manager.cc:359] Metadata directory not provided
I20260929 01:03:02.898108 10320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:03:02.898339 10320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:03:02.900557 10320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "dd55998a5a2347a4a1a8e52b12cf28fc"
format_stamp: "Formatted at 2026-09-29 01:03:02 on dist-test-slave-7zdl"
server_key: "82b8674d29de8b457dd5be8555d6e64f"
server_key_iv: "ae8c9d3d3f0d8d9431315990f0af0763"
server_key_version: "encryptionkey@0"
I20260929 01:03:02.901129 10320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dd55998a5a2347a4a1a8e52b12cf28fc"
format_stamp: "Formatted at 2026-09-29 01:03:02 on dist-test-slave-7zdl"
server_key: "82b8674d29de8b457dd5be8555d6e64f"
server_key_iv: "ae8c9d3d3f0d8d9431315990f0af0763"
server_key_version: "encryptionkey@0"
I20260929 01:03:02.904887 10320 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 01:03:02.907230 10334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:02.908402 10320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:03:02.908535 10320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "dd55998a5a2347a4a1a8e52b12cf28fc"
format_stamp: "Formatted at 2026-09-29 01:03:02 on dist-test-slave-7zdl"
server_key: "82b8674d29de8b457dd5be8555d6e64f"
server_key_iv: "ae8c9d3d3f0d8d9431315990f0af0763"
server_key_version: "encryptionkey@0"
I20260929 01:03:02.908644 10320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:03:02.913213 10320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:03:02.913841 10320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:03:02.914022 10320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:03:02.914601 10320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:03:02.915581 10320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:03:02.915652 10320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:02.915723 10320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:03:02.915766 10320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:02.924310 10320 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43309
I20260929 01:03:02.924345 10447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43309 every 8 connection(s)
I20260929 01:03:02.925215 10320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:03:02.925432 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 10320
I20260929 01:03:02.925556 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:03:02.925876 392 external_mini_cluster.cc:1428] Setting key a8924d6703f4a16f57ff94af7ffccc65
I20260929 01:03:02.927861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33777
--builtin_ntp_servers=127.0.98.20:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 01:03:02.931854 10168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:02.927202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50401 (local address 127.0.98.62:33777)
0929 01:03:02.927330 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:02.927334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:02.928455 (+ 1121us) server_negotiation.cc:407] Connection header received
0929 01:03:02.930021 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:02.930023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:02.930060 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:02.930128 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:02.931362 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:02.931365 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:02.931367 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:03:02.931378 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:02.931397 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:03:02.931405 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:02.931475 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:02.931642 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 01:03:02.931676 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 01:03:02.932960 10450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:02.927609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33777 (local address 127.0.98.2:50401)
0929 01:03:02.928297 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:02.928345 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:02.929792 (+ 1447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:02.930332 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:02.930343 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:02.930454 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:03:02.931061 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:03:02.931065 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:03:02.931119 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:03:02.931205 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:03:02.931491 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:03:02.931502 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:03:02.931512 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 01:03:02.931684 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 01:03:02.931935 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":130,"threads_started":1}
I20260929 01:03:02.934114 10448 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33777
I20260929 01:03:02.934341 10448 heartbeater.cc:464] Registering TS with master...
I20260929 01:03:02.934715 10448 heartbeater.cc:511] Master 127.0.98.62:33777 requested a full tablet report, sending...
I20260929 01:03:02.935726 10129 ts_manager.cc:194] Registered new tserver with Master: dd55998a5a2347a4a1a8e52b12cf28fc (127.0.98.2:43309)
W20260929 01:03:03.038242 10451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:03:03.038539 10451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:03:03.038641 10451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:03:03.042761 10451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:03:03.042858 10451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:03:03.042949 10451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:03:03.048108 10451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:03:03.049089 10451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:03:03.050384 10451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:03:03.059595 10459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:03.059782 10451 server_base.cc:1034] running on GCE node
W20260929 01:03:03.059608 10457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:03.059639 10456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:03.060273 10451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:03:03.060909 10451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:03:03.062085 10451 hybrid_clock.cc:648] HybridClock initialized: now 1790643783062071 us; error 27 us; skew 500 ppm
I20260929 01:03:03.063791 10451 webserver.cc:460] Webserver started at http://127.0.98.3:46389/ using document root <none> and password file <none>
I20260929 01:03:03.064400 10451 fs_manager.cc:359] Metadata directory not provided
I20260929 01:03:03.064486 10451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:03:03.064702 10451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:03:03.066630 10451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "f55f20918504454b9e319d728737e5ef"
format_stamp: "Formatted at 2026-09-29 01:03:03 on dist-test-slave-7zdl"
server_key: "53dfce242cf1e08b62d35a1460252c43"
server_key_iv: "3a4caa27d0c5e6ffb78ef412b4773283"
server_key_version: "encryptionkey@0"
I20260929 01:03:03.067265 10451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f55f20918504454b9e319d728737e5ef"
format_stamp: "Formatted at 2026-09-29 01:03:03 on dist-test-slave-7zdl"
server_key: "53dfce242cf1e08b62d35a1460252c43"
server_key_iv: "3a4caa27d0c5e6ffb78ef412b4773283"
server_key_version: "encryptionkey@0"
I20260929 01:03:03.070638 10451 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:03:03.072934 10465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:03.074016 10451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:03:03.074149 10451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "f55f20918504454b9e319d728737e5ef"
format_stamp: "Formatted at 2026-09-29 01:03:03 on dist-test-slave-7zdl"
server_key: "53dfce242cf1e08b62d35a1460252c43"
server_key_iv: "3a4caa27d0c5e6ffb78ef412b4773283"
server_key_version: "encryptionkey@0"
I20260929 01:03:03.074301 10451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:03:03.078783 10451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:03:03.079509 10451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:03:03.079711 10451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:03:03.080250 10451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:03:03.081157 10451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:03:03.081213 10451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:03.081274 10451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:03:03.081305 10451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:03.090055 10451 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33725
I20260929 01:03:03.090086 10578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33725 every 8 connection(s)
I20260929 01:03:03.090967 10451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:03:03.094661 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 10451
I20260929 01:03:03.094749 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:03:03.095039 392 external_mini_cluster.cc:1428] Setting key 79f5e40e06dbcaa148f9703e4a0f0669
I20260929 01:03:03.097174 10168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:03.092671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46587 (local address 127.0.98.62:33777)
0929 01:03:03.092809 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:03.092814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:03.093881 (+ 1067us) server_negotiation.cc:407] Connection header received
0929 01:03:03.095288 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:03.095292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:03.095345 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:03.095459 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:03.096675 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:03.096679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:03.096682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:03:03.096693 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:03.096714 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:03:03.096723 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:03.096824 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:03.096991 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 01:03:03.097032 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 01:03:03.097980 10581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:03.093136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33777 (local address 127.0.98.3:46587)
0929 01:03:03.093738 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:03.093779 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:03.095044 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:03.095647 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:03.095654 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:03.095782 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:03:03.096425 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:03:03.096427 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:03:03.096467 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:03:03.096532 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:03:03.096812 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:03:03.096827 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:03:03.096835 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 01:03:03.097032 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 01:03:03.097275 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":103,"threads_started":1}
I20260929 01:03:03.099040 10579 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33777
I20260929 01:03:03.099267 10579 heartbeater.cc:464] Registering TS with master...
I20260929 01:03:03.099640 10579 heartbeater.cc:511] Master 127.0.98.62:33777 requested a full tablet report, sending...
I20260929 01:03:03.100559 10129 ts_manager.cc:194] Registered new tserver with Master: f55f20918504454b9e319d728737e5ef (127.0.98.3:33725)
I20260929 01:03:03.108623 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 01:03:03.111395 10168 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:03:03.110480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47816 (local address 127.0.98.62:33777)
0929 01:03:03.110625 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:03.110629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:03.110846 (+ 217us) server_negotiation.cc:407] Connection header received
0929 01:03:03.110961 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:03.110963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:03.110995 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:03.111055 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:03.111268 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47816: BlockingRecv error: recv got EOF from 127.0.0.1:47816 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:03:03.111488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 10189
I20260929 01:03:03.117023 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 10320
I20260929 01:03:03.122197 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 10451
I20260929 01:03:03.127257 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 10098
2026-09-29T01:03:03Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (788 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:03:03 dist-test-slave-7zdl krb5kdc[10589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:03:03 dist-test-slave-7zdl krb5kdc[10589](info): set up 2 sockets
Sep 29 01:03:03 dist-test-slave-7zdl krb5kdc[10589](info): commencing operation
krb5kdc: starting...
W20260929 01:03:04.167917 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:03:04 dist-test-slave-7zdl krb5kdc[10589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:03:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:03:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:03:04.285854 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36337
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:03:04.393034 10602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:03:04.393317 10602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:03:04.393392 10602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:03:04.397437 10602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:03:04.397519 10602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:03:04.397557 10602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:03:04.397614 10602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:03:04.397655 10602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:03:04.402776 10602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:03:04.403857 10602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:03:04.404860 10602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:03:04.413307 10607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:04.413327 10608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:04.413373 10602 server_base.cc:1034] running on GCE node
W20260929 01:03:04.413365 10610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:04.414085 10602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:03:04.415232 10602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:03:04.416420 10602 hybrid_clock.cc:648] HybridClock initialized: now 1790643784416405 us; error 28 us; skew 500 ppm
Sep 29 01:03:04 dist-test-slave-7zdl krb5kdc[10589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:03:04.419163 10602 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:03:04.419992 10602 webserver.cc:460] Webserver started at http://127.0.98.62:40037/ using document root <none> and password file <none>
I20260929 01:03:04.420540 10602 fs_manager.cc:359] Metadata directory not provided
I20260929 01:03:04.420624 10602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:03:04.420845 10602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:03:04.422765 10602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "9d6320c11d2342ad8d2882a341360d9e"
format_stamp: "Formatted at 2026-09-29 01:03:04 on dist-test-slave-7zdl"
server_key: "7b6d68b5234394fb47084614549ba92a"
server_key_iv: "2f294e1a309f349a2b7e7965fe957b90"
server_key_version: "encryptionkey@0"
I20260929 01:03:04.423357 10602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "9d6320c11d2342ad8d2882a341360d9e"
format_stamp: "Formatted at 2026-09-29 01:03:04 on dist-test-slave-7zdl"
server_key: "7b6d68b5234394fb47084614549ba92a"
server_key_iv: "2f294e1a309f349a2b7e7965fe957b90"
server_key_version: "encryptionkey@0"
I20260929 01:03:04.426894 10602 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:03:04.429229 10617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:04.430397 10602 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:03:04.430528 10602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "9d6320c11d2342ad8d2882a341360d9e"
format_stamp: "Formatted at 2026-09-29 01:03:04 on dist-test-slave-7zdl"
server_key: "7b6d68b5234394fb47084614549ba92a"
server_key_iv: "2f294e1a309f349a2b7e7965fe957b90"
server_key_version: "encryptionkey@0"
I20260929 01:03:04.430637 10602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:03:04.459772 10602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:03:04.462917 10602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:03:04.463129 10602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:03:04.469966 10602 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36337
I20260929 01:03:04.470247 10669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36337 every 8 connection(s)
I20260929 01:03:04.470979 10602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:03:04.471817 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 10602
I20260929 01:03:04.471900 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:03:04.472190 392 external_mini_cluster.cc:1428] Setting key 5147429f0969bed16d226c3e7eb18300
I20260929 01:03:04.474174 10670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:03:04.480645 10670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d6320c11d2342ad8d2882a341360d9e: Bootstrap starting.
Sep 29 01:03:04 dist-test-slave-7zdl krb5kdc[10589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:03:04.483347 10670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d6320c11d2342ad8d2882a341360d9e: Neither blocks nor log segments found. Creating new log.
I20260929 01:03:04.484229 10670 log.cc:824] T 00000000000000000000000000000000 P 9d6320c11d2342ad8d2882a341360d9e: Log is configured to *not* fsync() on all Append() calls
I20260929 01:03:04.486379 10670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d6320c11d2342ad8d2882a341360d9e: No bootstrap required, opened a new log
I20260929 01:03:04.486666 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:04.473678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38790 (local address 127.0.98.62:36337)
0929 01:03:04.474096 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:04.474110 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:04.474145 (+ 35us) server_negotiation.cc:407] Connection header received
0929 01:03:04.474886 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:04.474903 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:04.475218 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:04.475481 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:04.476367 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:04.477330 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:04.478449 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:04.478959 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:04.479069 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:04.481783 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:04.481789 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:04.481804 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:03:04.481846 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:04.483778 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:04.484189 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:04.484194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:04.484200 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:04.484335 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:04.484587 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:04.484593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:04.484596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:04.485033 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:04.485202 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:04.485461 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 01:03:04.485739 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":143,"threads_started":1}
I20260929 01:03:04.490141 10670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d6320c11d2342ad8d2882a341360d9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6320c11d2342ad8d2882a341360d9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } }
I20260929 01:03:04.490350 10670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d6320c11d2342ad8d2882a341360d9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:03:04.490386 10670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d6320c11d2342ad8d2882a341360d9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d6320c11d2342ad8d2882a341360d9e, State: Initialized, Role: FOLLOWER
I20260929 01:03:04.490772 10670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d6320c11d2342ad8d2882a341360d9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6320c11d2342ad8d2882a341360d9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } }
I20260929 01:03:04.490942 10670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d6320c11d2342ad8d2882a341360d9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:03:04.490991 10670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d6320c11d2342ad8d2882a341360d9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:03:04.491060 10670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d6320c11d2342ad8d2882a341360d9e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:03:04.491871 10670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d6320c11d2342ad8d2882a341360d9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6320c11d2342ad8d2882a341360d9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } }
I20260929 01:03:04.492229 10670 leader_election.cc:302] T 00000000000000000000000000000000 P 9d6320c11d2342ad8d2882a341360d9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d6320c11d2342ad8d2882a341360d9e; no voters:
I20260929 01:03:04.492590 10670 leader_election.cc:288] T 00000000000000000000000000000000 P 9d6320c11d2342ad8d2882a341360d9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:03:04.492763 10675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d6320c11d2342ad8d2882a341360d9e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:03:04.492966 10675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d6320c11d2342ad8d2882a341360d9e [term 1 LEADER]: Becoming Leader. State: Replica: 9d6320c11d2342ad8d2882a341360d9e, State: Running, Role: LEADER
I20260929 01:03:04.493239 10675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d6320c11d2342ad8d2882a341360d9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6320c11d2342ad8d2882a341360d9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } }
I20260929 01:03:04.493775 10670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d6320c11d2342ad8d2882a341360d9e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:03:04.494936 10676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6320c11d2342ad8d2882a341360d9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d6320c11d2342ad8d2882a341360d9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6320c11d2342ad8d2882a341360d9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } } }
I20260929 01:03:04.494966 10677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6320c11d2342ad8d2882a341360d9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d6320c11d2342ad8d2882a341360d9e. Latest consensus state: current_term: 1 leader_uuid: "9d6320c11d2342ad8d2882a341360d9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6320c11d2342ad8d2882a341360d9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } } }
I20260929 01:03:04.495069 10677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6320c11d2342ad8d2882a341360d9e [sys.catalog]: This master's current role is: LEADER
I20260929 01:03:04.495069 10676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6320c11d2342ad8d2882a341360d9e [sys.catalog]: This master's current role is: LEADER
I20260929 01:03:04.495443 10684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:03:04.498498 10684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:03:04.504475 10684 catalog_manager.cc:1347] Generated new cluster ID: 217836e235f645e782a210714e2c3927
I20260929 01:03:04.504539 10684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:03:04.529342 10684 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:03:04.530280 10684 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:03:04.541895 10684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d6320c11d2342ad8d2882a341360d9e: Generated new TSK 0
I20260929 01:03:04.542582 10684 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 01:03:04.575558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36337
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:03:04.715031 10696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:03:04.715363 10696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:03:04.715442 10696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:03:04.719695 10696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:03:04.719781 10696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:03:04.719870 10696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:03:04.725201 10696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:03:04.726368 10696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:03:04.727480 10696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:03:04.737511 10704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:04.737625 10701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:04.737663 10696 server_base.cc:1034] running on GCE node
W20260929 01:03:04.737625 10702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:04.738379 10696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:03:04.739058 10696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:03:04.740216 10696 hybrid_clock.cc:648] HybridClock initialized: now 1790643784740201 us; error 29 us; skew 500 ppm
Sep 29 01:03:04 dist-test-slave-7zdl krb5kdc[10589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:03:04.742919 10696 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:03:04.743682 10696 webserver.cc:460] Webserver started at http://127.0.98.1:42443/ using document root <none> and password file <none>
I20260929 01:03:04.744194 10696 fs_manager.cc:359] Metadata directory not provided
I20260929 01:03:04.744251 10696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:03:04.744407 10696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:03:04.746357 10696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "4119ef73007545a6a8367b9548023394"
format_stamp: "Formatted at 2026-09-29 01:03:04 on dist-test-slave-7zdl"
server_key: "59be541746694023377e2d606e4edc24"
server_key_iv: "03f07a05757b9030662baa254d507487"
server_key_version: "encryptionkey@0"
I20260929 01:03:04.746929 10696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4119ef73007545a6a8367b9548023394"
format_stamp: "Formatted at 2026-09-29 01:03:04 on dist-test-slave-7zdl"
server_key: "59be541746694023377e2d606e4edc24"
server_key_iv: "03f07a05757b9030662baa254d507487"
server_key_version: "encryptionkey@0"
I20260929 01:03:04.750389 10696 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:03:04.752642 10711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:04.753809 10696 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:03:04.753925 10696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "4119ef73007545a6a8367b9548023394"
format_stamp: "Formatted at 2026-09-29 01:03:04 on dist-test-slave-7zdl"
server_key: "59be541746694023377e2d606e4edc24"
server_key_iv: "03f07a05757b9030662baa254d507487"
server_key_version: "encryptionkey@0"
I20260929 01:03:04.754020 10696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:03:04.770074 10696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:03:04.772976 10696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:03:04.773164 10696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:03:04.773717 10696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:03:04.774499 10696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:03:04.774544 10696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:04.774609 10696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:03:04.774639 10696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:04.783771 10696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46637
I20260929 01:03:04.783789 10824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46637 every 8 connection(s)
I20260929 01:03:04.784617 10696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:03:04.792191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 10696
I20260929 01:03:04.792279 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:03:04.792544 392 external_mini_cluster.cc:1428] Setting key 73947e3d6c436a091d54074a4464f60e
Sep 29 01:03:04 dist-test-slave-7zdl krb5kdc[10589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:03:04.799109 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:04.786300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51941 (local address 127.0.98.62:36337)
0929 01:03:04.786443 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:04.786447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:04.787267 (+ 820us) server_negotiation.cc:407] Connection header received
0929 01:03:04.788562 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:04.788566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:04.788623 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:04.788698 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:04.789835 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:04.790382 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:04.791584 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:04.791838 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:04.791954 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:04.794955 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:04.794959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:04.794962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:03:04.794989 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:04.796855 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:04.797365 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:04.797371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:04.797376 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:04.797470 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:04.797779 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:04.797785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:04.797790 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:04.798049 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:04.798162 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:04.798701 (+ 539us) server_negotiation.cc:299] Negotiation successful
0929 01:03:04.798883 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 01:03:04.799729 10827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:04.786652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36337 (local address 127.0.98.1:51941)
0929 01:03:04.787125 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:04.787166 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:04.788348 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:04.788864 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:04.788871 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:04.789215 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:03:04.789682 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:04.789695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:04.790508 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:04.790513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:04.791457 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:04.791467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:04.791990 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:04.791994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:04.792175 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:04.792822 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:03:04.792838 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:03:04.794761 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:03:04.796980 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:03:04.796985 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:03:04.796995 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:03:04.797239 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:03:04.797557 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:03:04.797559 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:03:04.797560 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:03:04.797680 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:03:04.798150 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:03:04.798155 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:03:04.798447 (+ 292us) client_negotiation.cc:769] Sending connection context
0929 01:03:04.798590 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 01:03:04.798894 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":157,"threads_started":1}
I20260929 01:03:04.800952 10825 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36337
I20260929 01:03:04.801218 10825 heartbeater.cc:464] Registering TS with master...
I20260929 01:03:04.801805 10825 heartbeater.cc:511] Master 127.0.98.62:36337 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 01:03:04.803388 10634 ts_manager.cc:194] Registered new tserver with Master: 4119ef73007545a6a8367b9548023394 (127.0.98.1:46637)
I20260929 01:03:04.804512 10634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 01:03:04.821346 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36337
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:03:04.930060 10830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:03:04.930351 10830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:03:04.930423 10830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:03:04.934469 10830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:03:04.934544 10830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:03:04.934626 10830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:03:04.939783 10830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:03:04.940761 10830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:03:04.941754 10830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:03:04.951263 10838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:04.951220 10835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:04.951185 10830 server_base.cc:1034] running on GCE node
W20260929 01:03:04.951195 10836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:04.951790 10830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:03:04.952385 10830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:03:04.953572 10830 hybrid_clock.cc:648] HybridClock initialized: now 1790643784953558 us; error 28 us; skew 500 ppm
Sep 29 01:03:04 dist-test-slave-7zdl krb5kdc[10589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:03:04.956323 10830 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:03:04.957186 10830 webserver.cc:460] Webserver started at http://127.0.98.2:44985/ using document root <none> and password file <none>
I20260929 01:03:04.957803 10830 fs_manager.cc:359] Metadata directory not provided
I20260929 01:03:04.957893 10830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:03:04.958097 10830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:03:04.960108 10830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "67f65798bf6b4117b660eb43903c5494"
format_stamp: "Formatted at 2026-09-29 01:03:04 on dist-test-slave-7zdl"
server_key: "b017c31e46b1ce65a4f9761b273ba60c"
server_key_iv: "0de56793364d286d0608065df006382f"
server_key_version: "encryptionkey@0"
I20260929 01:03:04.960654 10830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "67f65798bf6b4117b660eb43903c5494"
format_stamp: "Formatted at 2026-09-29 01:03:04 on dist-test-slave-7zdl"
server_key: "b017c31e46b1ce65a4f9761b273ba60c"
server_key_iv: "0de56793364d286d0608065df006382f"
server_key_version: "encryptionkey@0"
I20260929 01:03:04.964469 10830 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 01:03:04.966904 10845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:04.968097 10830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 01:03:04.968223 10830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "67f65798bf6b4117b660eb43903c5494"
format_stamp: "Formatted at 2026-09-29 01:03:04 on dist-test-slave-7zdl"
server_key: "b017c31e46b1ce65a4f9761b273ba60c"
server_key_iv: "0de56793364d286d0608065df006382f"
server_key_version: "encryptionkey@0"
I20260929 01:03:04.968338 10830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:03:04.985376 10830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:03:04.988763 10830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:03:04.988957 10830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:03:04.989534 10830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:03:04.990324 10830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:03:04.990389 10830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:04.990458 10830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:03:04.990509 10830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:05.000378 10830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41881
I20260929 01:03:05.000398 10958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41881 every 8 connection(s)
I20260929 01:03:05.001264 10830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:03:05.007565 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 10830
I20260929 01:03:05.007642 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:03:05.007917 392 external_mini_cluster.cc:1428] Setting key 9a3de9346c9be44f8ed35c310d118c26
Sep 29 01:03:05 dist-test-slave-7zdl krb5kdc[10589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:03:05.015249 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:05.002983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35619 (local address 127.0.98.62:36337)
0929 01:03:05.003121 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:05.003125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:05.003932 (+ 807us) server_negotiation.cc:407] Connection header received
0929 01:03:05.005226 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:05.005230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:05.005289 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:05.005358 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:05.006521 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.007119 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.008404 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.008669 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.008804 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:05.011369 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:05.011375 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:05.011379 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:03:05.011412 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:05.013196 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:05.013699 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:05.013704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:05.013706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:05.013794 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:05.014138 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:05.014142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:05.014144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:05.014334 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:05.014444 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:05.014973 (+ 529us) server_negotiation.cc:299] Negotiation successful
0929 01:03:05.015090 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 01:03:05.016077 10961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:05.003360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36337 (local address 127.0.98.2:35619)
0929 01:03:05.003775 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:05.003817 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:05.005000 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:05.005534 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:05.005541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:05.005900 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:03:05.006370 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:05.006384 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.007248 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.007254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:05.008274 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:05.008286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.008855 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.008858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:05.008999 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:05.009682 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:03:05.009698 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:03:05.011191 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:03:05.013307 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:03:05.013314 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:03:05.013326 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:03:05.013574 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:03:05.013899 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:03:05.013904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:03:05.013911 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:03:05.014039 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:03:05.014439 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:03:05.014446 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:03:05.014684 (+ 238us) client_negotiation.cc:769] Sending connection context
0929 01:03:05.014931 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 01:03:05.015254 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":95,"threads_started":1}
I20260929 01:03:05.017217 10959 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36337
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 01:03:05.017472 10959 heartbeater.cc:464] Registering TS with master...
I20260929 01:03:05.017937 10959 heartbeater.cc:511] Master 127.0.98.62:36337 requested a full tablet report, sending...
I20260929 01:03:05.019116 10634 ts_manager.cc:194] Registered new tserver with Master: 67f65798bf6b4117b660eb43903c5494 (127.0.98.2:41881)
I20260929 01:03:05.019639 10634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 01:03:05.033845 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36337
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:03:05.142762 10964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:03:05.143061 10964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:03:05.143110 10964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:03:05.147073 10964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:03:05.147143 10964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:03:05.147245 10964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:03:05.152551 10964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:03:05.153564 10964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:03:05.154640 10964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:03:05.164584 10972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:05.164588 10969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:05.164636 10964 server_base.cc:1034] running on GCE node
W20260929 01:03:05.164598 10970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:05.165212 10964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:03:05.165855 10964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:03:05.167030 10964 hybrid_clock.cc:648] HybridClock initialized: now 1790643785166997 us; error 46 us; skew 500 ppm
Sep 29 01:03:05 dist-test-slave-7zdl krb5kdc[10589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:03:05.169847 10964 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:03:05.170746 10964 webserver.cc:460] Webserver started at http://127.0.98.3:34135/ using document root <none> and password file <none>
I20260929 01:03:05.171339 10964 fs_manager.cc:359] Metadata directory not provided
I20260929 01:03:05.171427 10964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:03:05.171677 10964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:03:05.173732 10964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "8d38847788174fdd9e8c263c513c8757"
format_stamp: "Formatted at 2026-09-29 01:03:05 on dist-test-slave-7zdl"
server_key: "84f43c2bccc4240a0b00e36e59f90c08"
server_key_iv: "89a3b799960a456a0af03a182f32334b"
server_key_version: "encryptionkey@0"
I20260929 01:03:05.174278 10964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8d38847788174fdd9e8c263c513c8757"
format_stamp: "Formatted at 2026-09-29 01:03:05 on dist-test-slave-7zdl"
server_key: "84f43c2bccc4240a0b00e36e59f90c08"
server_key_iv: "89a3b799960a456a0af03a182f32334b"
server_key_version: "encryptionkey@0"
I20260929 01:03:05.177938 10964 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 01:03:05.180318 10979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:05.181531 10964 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:03:05.181681 10964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "8d38847788174fdd9e8c263c513c8757"
format_stamp: "Formatted at 2026-09-29 01:03:05 on dist-test-slave-7zdl"
server_key: "84f43c2bccc4240a0b00e36e59f90c08"
server_key_iv: "89a3b799960a456a0af03a182f32334b"
server_key_version: "encryptionkey@0"
I20260929 01:03:05.181790 10964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:03:05.205042 10964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:03:05.208292 10964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:03:05.208508 10964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:03:05.209098 10964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:03:05.209932 10964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:03:05.210003 10964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:05.210073 10964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:03:05.210122 10964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:05.219013 10964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45271
I20260929 01:03:05.219031 11092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45271 every 8 connection(s)
I20260929 01:03:05.219882 10964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
Sep 29 01:03:05 dist-test-slave-7zdl krb5kdc[10589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:03:05.229863 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 10964
I20260929 01:03:05.230010 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:03:05.230435 392 external_mini_cluster.cc:1428] Setting key aede1601e6ee0e20212ac94473d32622
I20260929 01:03:05.233592 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:05.221851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43479 (local address 127.0.98.62:36337)
0929 01:03:05.221983 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:05.221987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:05.223199 (+ 1212us) server_negotiation.cc:407] Connection header received
0929 01:03:05.224110 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:05.224113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:05.224172 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:05.224288 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:05.225387 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.225917 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.227101 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.227327 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.227440 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:05.229821 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:05.229830 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:05.229832 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:03:05.229856 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:05.231603 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:05.232179 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:05.232183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:05.232185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:05.232280 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:05.232696 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:05.232698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:05.232699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:05.232867 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:05.232940 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:05.233393 (+ 453us) server_negotiation.cc:299] Negotiation successful
0929 01:03:05.233482 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 01:03:05.234463 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:05.222210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36337 (local address 127.0.98.3:43479)
0929 01:03:05.223064 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:05.223102 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:05.223905 (+ 803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:05.224452 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:05.224458 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:05.224805 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:03:05.225250 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:05.225263 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.226044 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.226048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:05.226984 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:05.226993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.227474 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.227478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:05.227640 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:05.228224 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:03:05.228241 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:03:05.229652 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:03:05.231758 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:03:05.231765 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:03:05.231781 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:03:05.232084 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:03:05.232376 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:03:05.232378 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:03:05.232379 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:03:05.232573 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:03:05.232954 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:03:05.232958 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:03:05.233191 (+ 233us) client_negotiation.cc:769] Sending connection context
0929 01:03:05.233349 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 01:03:05.233619 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":539,"threads_started":1}
I20260929 01:03:05.235579 11093 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36337
I20260929 01:03:05.235821 11093 heartbeater.cc:464] Registering TS with master...
I20260929 01:03:05.236299 11093 heartbeater.cc:511] Master 127.0.98.62:36337 requested a full tablet report, sending...
I20260929 01:03:05.237211 10634 ts_manager.cc:194] Registered new tserver with Master: 8d38847788174fdd9e8c263c513c8757 (127.0.98.3:45271)
I20260929 01:03:05.237767 10634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43479
I20260929 01:03:05.244134 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:03:05.253324 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:05.245937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38800 (local address 127.0.98.62:36337)
0929 01:03:05.246065 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:05.246069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:05.246238 (+ 169us) server_negotiation.cc:407] Connection header received
0929 01:03:05.246400 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:05.246403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:05.246468 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:05.246570 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:05.247342 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.247860 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.248683 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.248905 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.249013 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:05.250332 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:05.250336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:05.250338 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:03:05.250369 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:05.251671 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:05.252081 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:05.252086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:05.252087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:05.252192 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:05.252483 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:05.252486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:05.252487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:05.252688 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:05.252837 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:05.253049 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 01:03:05.253157 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 01:03:05.256071 10634 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:03:05.259212 10634 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:03:05.272917 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:05.267393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35556 (local address 127.0.98.1:46637)
0929 01:03:05.267649 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:05.267672 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:05.267693 (+ 21us) server_negotiation.cc:407] Connection header received
0929 01:03:05.267899 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:05.267905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:05.268126 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:05.268653 (+ 527us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:03:05.268832 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.269955 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.271534 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.272205 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.272330 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:05.272478 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:05.272618 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 01:03:05.272771 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":96,"threads_started":1}
I20260929 01:03:05.272902 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:05.267260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46637 (local address 127.0.0.1:35556)
0929 01:03:05.267564 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:05.267606 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:05.267716 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:05.268313 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:05.268319 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:05.268346 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:03:05.268493 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:05.268501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.270100 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.270104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:05.271422 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:05.271429 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.272327 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.272331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:05.272440 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:05.272475 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 01:03:05.272587 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 01:03:05.272756 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":4,"thread_start_us":90,"threads_started":1}
I20260929 01:03:05.273468 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:05.267967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41881 (local address 127.0.0.1:59762)
0929 01:03:05.268200 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:05.268215 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:05.268291 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:05.269341 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:05.269344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:05.269360 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:03:05.269457 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:05.269464 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.270736 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.270748 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:05.271921 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:05.271930 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.273103 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.273107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:05.273192 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:05.273214 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 01:03:05.273262 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 01:03:05.273336 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":18,"thread_start_us":104,"threads_started":1}
I20260929 01:03:05.274083 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:05.268158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59762 (local address 127.0.98.2:41881)
0929 01:03:05.268958 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:05.268972 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:05.268995 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:03:05.269074 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:05.269078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:05.269209 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:05.269364 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:03:05.269582 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.270573 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.272049 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.272981 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.273648 (+ 667us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:05.273791 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:05.273842 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 01:03:05.273943 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":112,"threads_started":1}
I20260929 01:03:05.274466 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:05.268669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.0.98.3:45271)
0929 01:03:05.269109 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:05.269120 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:05.269142 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:03:05.269211 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:05.269216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:05.269409 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:05.269549 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:03:05.269721 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.270564 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.273026 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.273841 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.273945 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:05.274064 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:05.274209 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 01:03:05.274309 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":157,"threads_started":1}
I20260929 01:03:05.275107 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:05.268296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45271 (local address 127.0.0.1:44986)
0929 01:03:05.268493 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:05.268507 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:05.268579 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:05.269543 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:05.269548 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:05.269565 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:03:05.269637 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:05.269642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.270691 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.270698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:05.272871 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:05.272890 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.273961 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.273964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:05.274048 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:05.274069 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 01:03:05.274144 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 01:03:05.274984 (+ 840us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":57,"thread_start_us":90,"threads_started":1}
I20260929 01:03:05.276088 10759 tablet_service.cc:1467] Processing CreateTablet for tablet a3a8e78327c046558bf84885c3efb0ae (DEFAULT_TABLE table=test-table [id=13a7394dd7ec41f3b0e5ac4b1ca75f5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:03:05.276625 10888 tablet_service.cc:1467] Processing CreateTablet for tablet a3a8e78327c046558bf84885c3efb0ae (DEFAULT_TABLE table=test-table [id=13a7394dd7ec41f3b0e5ac4b1ca75f5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:03:05.277207 10759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a8e78327c046558bf84885c3efb0ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:03:05.277537 10888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a8e78327c046558bf84885c3efb0ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:03:05.277796 11027 tablet_service.cc:1467] Processing CreateTablet for tablet a3a8e78327c046558bf84885c3efb0ae (DEFAULT_TABLE table=test-table [id=13a7394dd7ec41f3b0e5ac4b1ca75f5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:03:05.278795 11027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3a8e78327c046558bf84885c3efb0ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:03:05.283358 11110 tablet_bootstrap.cc:492] T a3a8e78327c046558bf84885c3efb0ae P 67f65798bf6b4117b660eb43903c5494: Bootstrap starting.
I20260929 01:03:05.285017 11111 tablet_bootstrap.cc:492] T a3a8e78327c046558bf84885c3efb0ae P 4119ef73007545a6a8367b9548023394: Bootstrap starting.
I20260929 01:03:05.285220 11112 tablet_bootstrap.cc:492] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757: Bootstrap starting.
I20260929 01:03:05.287047 11111 tablet_bootstrap.cc:654] T a3a8e78327c046558bf84885c3efb0ae P 4119ef73007545a6a8367b9548023394: Neither blocks nor log segments found. Creating new log.
I20260929 01:03:05.287182 11110 tablet_bootstrap.cc:654] T a3a8e78327c046558bf84885c3efb0ae P 67f65798bf6b4117b660eb43903c5494: Neither blocks nor log segments found. Creating new log.
I20260929 01:03:05.287318 11112 tablet_bootstrap.cc:654] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757: Neither blocks nor log segments found. Creating new log.
I20260929 01:03:05.288085 11111 log.cc:824] T a3a8e78327c046558bf84885c3efb0ae P 4119ef73007545a6a8367b9548023394: Log is configured to *not* fsync() on all Append() calls
I20260929 01:03:05.288113 11110 log.cc:824] T a3a8e78327c046558bf84885c3efb0ae P 67f65798bf6b4117b660eb43903c5494: Log is configured to *not* fsync() on all Append() calls
I20260929 01:03:05.288422 11112 log.cc:824] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757: Log is configured to *not* fsync() on all Append() calls
I20260929 01:03:05.290037 11111 tablet_bootstrap.cc:492] T a3a8e78327c046558bf84885c3efb0ae P 4119ef73007545a6a8367b9548023394: No bootstrap required, opened a new log
I20260929 01:03:05.290037 11110 tablet_bootstrap.cc:492] T a3a8e78327c046558bf84885c3efb0ae P 67f65798bf6b4117b660eb43903c5494: No bootstrap required, opened a new log
I20260929 01:03:05.290225 11111 ts_tablet_manager.cc:1397] T a3a8e78327c046558bf84885c3efb0ae P 4119ef73007545a6a8367b9548023394: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 01:03:05.290225 11110 ts_tablet_manager.cc:1397] T a3a8e78327c046558bf84885c3efb0ae P 67f65798bf6b4117b660eb43903c5494: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 01:03:05.290313 11112 tablet_bootstrap.cc:492] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757: No bootstrap required, opened a new log
I20260929 01:03:05.290482 11112 ts_tablet_manager.cc:1397] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 01:03:05.293217 11112 raft_consensus.cc:348] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4119ef73007545a6a8367b9548023394" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46637 } } peers { permanent_uuid: "67f65798bf6b4117b660eb43903c5494" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41881 } } peers { permanent_uuid: "8d38847788174fdd9e8c263c513c8757" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260929 01:03:05.293509 11112 raft_consensus.cc:374] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:03:05.293586 11112 raft_consensus.cc:729] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d38847788174fdd9e8c263c513c8757, State: Initialized, Role: FOLLOWER
I20260929 01:03:05.294112 11112 consensus_queue.cc:260] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4119ef73007545a6a8367b9548023394" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46637 } } peers { permanent_uuid: "67f65798bf6b4117b660eb43903c5494" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41881 } } peers { permanent_uuid: "8d38847788174fdd9e8c263c513c8757" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260929 01:03:05.294042 11111 raft_consensus.cc:348] T a3a8e78327c046558bf84885c3efb0ae P 4119ef73007545a6a8367b9548023394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4119ef73007545a6a8367b9548023394" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46637 } } peers { permanent_uuid: "67f65798bf6b4117b660eb43903c5494" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41881 } } peers { permanent_uuid: "8d38847788174fdd9e8c263c513c8757" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260929 01:03:05.294042 11110 raft_consensus.cc:348] T a3a8e78327c046558bf84885c3efb0ae P 67f65798bf6b4117b660eb43903c5494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4119ef73007545a6a8367b9548023394" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46637 } } peers { permanent_uuid: "67f65798bf6b4117b660eb43903c5494" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41881 } } peers { permanent_uuid: "8d38847788174fdd9e8c263c513c8757" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260929 01:03:05.294301 11110 raft_consensus.cc:374] T a3a8e78327c046558bf84885c3efb0ae P 67f65798bf6b4117b660eb43903c5494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:03:05.294313 11111 raft_consensus.cc:374] T a3a8e78327c046558bf84885c3efb0ae P 4119ef73007545a6a8367b9548023394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:03:05.294370 11111 raft_consensus.cc:729] T a3a8e78327c046558bf84885c3efb0ae P 4119ef73007545a6a8367b9548023394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4119ef73007545a6a8367b9548023394, State: Initialized, Role: FOLLOWER
I20260929 01:03:05.294458 11110 raft_consensus.cc:729] T a3a8e78327c046558bf84885c3efb0ae P 67f65798bf6b4117b660eb43903c5494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67f65798bf6b4117b660eb43903c5494, State: Initialized, Role: FOLLOWER
I20260929 01:03:05.294761 11093 heartbeater.cc:503] Master 127.0.98.62:36337 was elected leader, sending a full tablet report...
I20260929 01:03:05.294919 11110 consensus_queue.cc:260] T a3a8e78327c046558bf84885c3efb0ae P 67f65798bf6b4117b660eb43903c5494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4119ef73007545a6a8367b9548023394" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46637 } } peers { permanent_uuid: "67f65798bf6b4117b660eb43903c5494" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41881 } } peers { permanent_uuid: "8d38847788174fdd9e8c263c513c8757" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260929 01:03:05.294919 11111 consensus_queue.cc:260] T a3a8e78327c046558bf84885c3efb0ae P 4119ef73007545a6a8367b9548023394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4119ef73007545a6a8367b9548023394" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46637 } } peers { permanent_uuid: "67f65798bf6b4117b660eb43903c5494" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41881 } } peers { permanent_uuid: "8d38847788174fdd9e8c263c513c8757" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260929 01:03:05.295148 11112 ts_tablet_manager.cc:1428] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 01:03:05.295846 11111 ts_tablet_manager.cc:1428] T a3a8e78327c046558bf84885c3efb0ae P 4119ef73007545a6a8367b9548023394: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 01:03:05.295846 11110 ts_tablet_manager.cc:1428] T a3a8e78327c046558bf84885c3efb0ae P 67f65798bf6b4117b660eb43903c5494: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 01:03:05.296169 10825 heartbeater.cc:503] Master 127.0.98.62:36337 was elected leader, sending a full tablet report...
I20260929 01:03:05.296169 10959 heartbeater.cc:503] Master 127.0.98.62:36337 was elected leader, sending a full tablet report...
I20260929 01:03:05.398540 11116 raft_consensus.cc:482] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:03:05.398742 11116 raft_consensus.cc:504] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4119ef73007545a6a8367b9548023394" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46637 } } peers { permanent_uuid: "67f65798bf6b4117b660eb43903c5494" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41881 } } peers { permanent_uuid: "8d38847788174fdd9e8c263c513c8757" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260929 01:03:05.399919 11116 leader_election.cc:288] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4119ef73007545a6a8367b9548023394 (127.0.98.1:46637), 67f65798bf6b4117b660eb43903c5494 (127.0.98.2:41881)
I20260929 01:03:05.403508 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:05.400169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46637 (local address 127.0.98.3:56927)
0929 01:03:05.400338 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:05.400356 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:05.400471 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:05.400783 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:05.400786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:05.400803 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:03:05.400929 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:05.400935 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.401658 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.401663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:05.402430 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:05.402439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.403081 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.403085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:05.403189 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:05.403218 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 01:03:05.403304 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 01:03:05.403379 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 01:03:05.403508 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:05.400249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56927 (local address 127.0.98.1:46637)
0929 01:03:05.400389 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:05.400395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:05.400412 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:03:05.400607 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:05.400614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:05.400673 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:05.400755 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:03:05.401043 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.401544 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.402557 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.402963 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.403084 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:05.403149 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:05.403337 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 01:03:05.403403 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 01:03:05.404023 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:05.400472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41881 (local address 127.0.98.3:40767)
0929 01:03:05.400858 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:05.400868 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:05.400975 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:05.401210 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:05.401213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:05.401228 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:03:05.401354 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:05.401358 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.402065 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.402070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:05.402902 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:05.402911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.403567 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.403571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:05.403677 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:05.403706 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 01:03:05.403810 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 01:03:05.403888 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":130,"threads_started":1}
I20260929 01:03:05.404013 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:05.400593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40767 (local address 127.0.98.2:41881)
0929 01:03:05.400731 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:05.400744 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:05.400946 (+ 202us) server_negotiation.cc:407] Connection header received
0929 01:03:05.401065 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:05.401068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:05.401118 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:05.401201 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:03:05.401468 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.401950 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.403017 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.403426 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.403538 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:05.403595 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:05.403848 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 01:03:05.403914 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 01:03:05.404157 10779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a8e78327c046558bf84885c3efb0ae" candidate_uuid: "8d38847788174fdd9e8c263c513c8757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4119ef73007545a6a8367b9548023394" is_pre_election: true
I20260929 01:03:05.404479 10779 raft_consensus.cc:2454] T a3a8e78327c046558bf84885c3efb0ae P 4119ef73007545a6a8367b9548023394 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d38847788174fdd9e8c263c513c8757 in term 0.
I20260929 01:03:05.404548 10901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a8e78327c046558bf84885c3efb0ae" candidate_uuid: "8d38847788174fdd9e8c263c513c8757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67f65798bf6b4117b660eb43903c5494" is_pre_election: true
I20260929 01:03:05.404844 10901 raft_consensus.cc:2454] T a3a8e78327c046558bf84885c3efb0ae P 67f65798bf6b4117b660eb43903c5494 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d38847788174fdd9e8c263c513c8757 in term 0.
I20260929 01:03:05.404973 10983 leader_election.cc:302] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4119ef73007545a6a8367b9548023394, 8d38847788174fdd9e8c263c513c8757; no voters:
I20260929 01:03:05.405257 11116 raft_consensus.cc:2781] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:03:05.405330 11116 raft_consensus.cc:482] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:03:05.405360 11116 raft_consensus.cc:3037] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:03:05.406399 11116 raft_consensus.cc:504] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4119ef73007545a6a8367b9548023394" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46637 } } peers { permanent_uuid: "67f65798bf6b4117b660eb43903c5494" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41881 } } peers { permanent_uuid: "8d38847788174fdd9e8c263c513c8757" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260929 01:03:05.406729 11116 leader_election.cc:288] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757 [CANDIDATE]: Term 1 election: Requested vote from peers 4119ef73007545a6a8367b9548023394 (127.0.98.1:46637), 67f65798bf6b4117b660eb43903c5494 (127.0.98.2:41881)
I20260929 01:03:05.407140 10901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a8e78327c046558bf84885c3efb0ae" candidate_uuid: "8d38847788174fdd9e8c263c513c8757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67f65798bf6b4117b660eb43903c5494"
I20260929 01:03:05.407178 10779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a8e78327c046558bf84885c3efb0ae" candidate_uuid: "8d38847788174fdd9e8c263c513c8757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4119ef73007545a6a8367b9548023394"
I20260929 01:03:05.407234 10901 raft_consensus.cc:3037] T a3a8e78327c046558bf84885c3efb0ae P 67f65798bf6b4117b660eb43903c5494 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:03:05.407325 10779 raft_consensus.cc:3037] T a3a8e78327c046558bf84885c3efb0ae P 4119ef73007545a6a8367b9548023394 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:03:05.408268 10901 raft_consensus.cc:2454] T a3a8e78327c046558bf84885c3efb0ae P 67f65798bf6b4117b660eb43903c5494 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d38847788174fdd9e8c263c513c8757 in term 1.
I20260929 01:03:05.408381 10779 raft_consensus.cc:2454] T a3a8e78327c046558bf84885c3efb0ae P 4119ef73007545a6a8367b9548023394 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d38847788174fdd9e8c263c513c8757 in term 1.
I20260929 01:03:05.408546 10983 leader_election.cc:302] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67f65798bf6b4117b660eb43903c5494, 8d38847788174fdd9e8c263c513c8757; no voters:
I20260929 01:03:05.408697 11116 raft_consensus.cc:2781] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:03:05.408957 11116 raft_consensus.cc:686] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757 [term 1 LEADER]: Becoming Leader. State: Replica: 8d38847788174fdd9e8c263c513c8757, State: Running, Role: LEADER
I20260929 01:03:05.409281 11116 consensus_queue.cc:237] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4119ef73007545a6a8367b9548023394" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46637 } } peers { permanent_uuid: "67f65798bf6b4117b660eb43903c5494" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41881 } } peers { permanent_uuid: "8d38847788174fdd9e8c263c513c8757" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } }
I20260929 01:03:05.412412 10633 catalog_manager.cc:5495] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d38847788174fdd9e8c263c513c8757 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8d38847788174fdd9e8c263c513c8757" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4119ef73007545a6a8367b9548023394" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67f65798bf6b4117b660eb43903c5494" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d38847788174fdd9e8c263c513c8757" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45271 } health_report { overall_health: HEALTHY } } }
I20260929 01:03:05.423338 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:05.420239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44992 (local address 127.0.98.3:45271)
0929 01:03:05.420356 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:05.420359 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:05.420371 (+ 12us) server_negotiation.cc:407] Connection header received
0929 01:03:05.420486 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:05.420488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:05.420538 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:05.420615 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:03:05.420807 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.421334 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.422115 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:05.422316 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:05.422388 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:05.422606 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:03:05.423045 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:03:05.423133 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:05.423187 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 01:03:05.423231 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 01:03:05.428960 10779 raft_consensus.cc:1261] T a3a8e78327c046558bf84885c3efb0ae P 4119ef73007545a6a8367b9548023394 [term 1 FOLLOWER]: Refusing update from remote peer 8d38847788174fdd9e8c263c513c8757: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:03:05.428960 10901 raft_consensus.cc:1261] T a3a8e78327c046558bf84885c3efb0ae P 67f65798bf6b4117b660eb43903c5494 [term 1 FOLLOWER]: Refusing update from remote peer 8d38847788174fdd9e8c263c513c8757: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:03:05.429700 11120 consensus_queue.cc:1035] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4119ef73007545a6a8367b9548023394" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:03:05.429924 11116 consensus_queue.cc:1035] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67f65798bf6b4117b660eb43903c5494" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:03:05.440330 11124 mvcc.cc:205] Tried to move back new op lower bound from 7334476945109708800 to 7334476945038843904. Current Snapshot: MvccSnapshot[applied={T|T < 7334476945109708800 or (T in {7334476945109708800})}]
I20260929 01:03:05.445206 10634 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:03:05.445350 10634 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:03:05.447754 10634 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d6320c11d2342ad8d2882a341360d9e: Sending DeleteTablet for 3 replicas of tablet a3a8e78327c046558bf84885c3efb0ae
I20260929 01:03:05.448603 11027 tablet_service.cc:1514] Processing DeleteTablet for tablet a3a8e78327c046558bf84885c3efb0ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:03:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44986
I20260929 01:03:05.448612 10759 tablet_service.cc:1514] Processing DeleteTablet for tablet a3a8e78327c046558bf84885c3efb0ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:03:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35556
I20260929 01:03:05.449069 10888 tablet_service.cc:1514] Processing DeleteTablet for tablet a3a8e78327c046558bf84885c3efb0ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:03:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59762
I20260929 01:03:05.449213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 10696
I20260929 01:03:05.449075 11134 tablet_replica.cc:330] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757: stopping tablet replica
I20260929 01:03:05.449548 11134 raft_consensus.cc:2229] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:03:05.449983 11134 raft_consensus.cc:2258] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:03:05.451514 11134 ts_tablet_manager.cc:1905] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:03:05.452997 11135 tablet_replica.cc:330] T a3a8e78327c046558bf84885c3efb0ae P 67f65798bf6b4117b660eb43903c5494: stopping tablet replica
I20260929 01:03:05.453367 11135 raft_consensus.cc:2229] T a3a8e78327c046558bf84885c3efb0ae P 67f65798bf6b4117b660eb43903c5494 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:03:05.453732 11135 raft_consensus.cc:2258] T a3a8e78327c046558bf84885c3efb0ae P 67f65798bf6b4117b660eb43903c5494 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:03:05.454993 11134 ts_tablet_manager.cc:1918] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:03:05.455139 11134 log.cc:1196] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/a3a8e78327c046558bf84885c3efb0ae
I20260929 01:03:05.455267 11135 ts_tablet_manager.cc:1905] T a3a8e78327c046558bf84885c3efb0ae P 67f65798bf6b4117b660eb43903c5494: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:03:05.455551 11134 ts_tablet_manager.cc:1939] T a3a8e78327c046558bf84885c3efb0ae P 8d38847788174fdd9e8c263c513c8757: Deleting consensus metadata
I20260929 01:03:05.457095 10621 catalog_manager.cc:4841] TS 8d38847788174fdd9e8c263c513c8757 (127.0.98.3:45271): tablet a3a8e78327c046558bf84885c3efb0ae (table test-table [id=13a7394dd7ec41f3b0e5ac4b1ca75f5b]) successfully deleted
W20260929 01:03:05.457723 10621 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46637 (error 108)
I20260929 01:03:05.457873 11135 ts_tablet_manager.cc:1918] T a3a8e78327c046558bf84885c3efb0ae P 67f65798bf6b4117b660eb43903c5494: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:03:05.457971 11135 log.cc:1196] T a3a8e78327c046558bf84885c3efb0ae P 67f65798bf6b4117b660eb43903c5494: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790643720886468-392-0/minicluster-data/ts-1/wal/wals/a3a8e78327c046558bf84885c3efb0ae
I20260929 01:03:05.458321 11135 ts_tablet_manager.cc:1939] T a3a8e78327c046558bf84885c3efb0ae P 67f65798bf6b4117b660eb43903c5494: Deleting consensus metadata
I20260929 01:03:05.458451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 10830
I20260929 01:03:05.459259 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:05.458831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46637 (local address 127.0.0.1:35572)
0929 01:03:05.459085 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:05.459158 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
W20260929 01:03:05.459419 10621 catalog_manager.cc:4584] TS 4119ef73007545a6a8367b9548023394 (127.0.98.1:46637): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a3a8e78327c046558bf84885c3efb0ae: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46637: connect: Connection refused (error 111)
I20260929 01:03:05.464718 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 10964
I20260929 01:03:05.466387 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:05.464940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41881 (local address 127.0.0.1:59772)
0929 01:03:05.466264 (+ 1324us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:05.466315 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1253}
I20260929 01:03:05.470456 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 10602
2026-09-29T01:03:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2344 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:03:05 dist-test-slave-7zdl krb5kdc[11139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:03:05 dist-test-slave-7zdl krb5kdc[11139](info): set up 2 sockets
Sep 29 01:03:05 dist-test-slave-7zdl krb5kdc[11139](info): commencing operation
krb5kdc: starting...
W20260929 01:03:06.543889 392 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:03:06 dist-test-slave-7zdl krb5kdc[11139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:03:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:03:06Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260929 01:03:06.707213 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskS2TZOM/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/postgres -l logfile start
2026-09-29 01:03:09.188 UTC [11169] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 01:03:09.188 UTC [11169] LOG: listening on IPv4 address "127.0.98.20", port 60559
2026-09-29 01:03:09.192 UTC [11169] LOG: listening on Unix socket "/tmp/.s.PGSQL.60559"
2026-09-29 01:03:09.214 UTC [11172] LOG: database system was shut down at 2026-09-29 01:03:07 UTC
2026-09-29 01:03:09.217 UTC [11169] LOG: database system is ready to accept connections
I20260929 01:03:10.175704 392 mini_postgres.cc:96] Postgres bound to 60559
2026-09-29 01:03:10.182 UTC [11180] FATAL: database "slave" does not exist
127.0.98.20:60559 - accepting connections
I20260929 01:03:10.182700 392 mini_ranger.cc:161] Starting Ranger...
I20260929 01:03:10.197188 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 01:03:10.673635 392 mini_ranger.cc:87] Created ranger Postgres database
I20260929 01:03:10.673732 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ranger-admin
2026-09-29 01:03:10,969 [I] DB FLAVOR :POSTGRES
2026-09-29 01:03:10,970 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 01:03:10,970 [I] Checking connection..
2026-09-29 01:03:10,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 01:03:11,200 [I] Checking connection passed.
2026-09-29 01:03:11,200 [I] --------- Verifying version history table ---------
2026-09-29 01:03:11,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 01:03:11,426 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 01:03:11,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 01:03:11,652 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 01:03:11,652 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 01:03:11,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60559/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 01:03:11.908 UTC [11244] WARNING: there is no transaction in progress
2026-09-29 01:03:11,914 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 01:03:11,915 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 01:03:12,165 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 01:03:12,165 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 01:03:12,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 01:03:12,410 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 01:03:12,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7zdl.c.gcp-upstream.internal','N') ;"
2026-09-29 01:03:12,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 01:03:13,136 [I] Table x_portal_user does not exist in database ranger
2026-09-29 01:03:13,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 01:03:13,380 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 01:03:13,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 01:03:13,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 01:03:13,881 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 01:03:13,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60559/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 01:03:14.840 UTC [11352] WARNING: there is no transaction in progress
2026-09-29 01:03:14.856 UTC [11352] WARNING: there is no transaction in progress
2026-09-29 01:03:14.872 UTC [11352] WARNING: there is no transaction in progress
2026-09-29 01:03:14.889 UTC [11352] WARNING: there is no transaction in progress
2026-09-29 01:03:14.905 UTC [11352] WARNING: there is no transaction in progress
2026-09-29 01:03:14.923 UTC [11352] WARNING: there is no transaction in progress
2026-09-29 01:03:14.987 UTC [11352] WARNING: there is no transaction in progress
2026-09-29 01:03:14.994 UTC [11352] WARNING: there is no transaction in progress
2026-09-29 01:03:15.009 UTC [11352] WARNING: there is no transaction in progress
2026-09-29 01:03:15.018 UTC [11352] WARNING: there is no transaction in progress
2026-09-29 01:03:15.028 UTC [11352] WARNING: there is no transaction in progress
2026-09-29 01:03:15.042 UTC [11352] WARNING: there is no transaction in progress
2026-09-29 01:03:15.052 UTC [11352] WARNING: there is no transaction in progress
2026-09-29 01:03:15.059 UTC [11352] WARNING: there is no transaction in progress
2026-09-29 01:03:15.066 UTC [11352] WARNING: there is no transaction in progress
2026-09-29 01:03:15.073 UTC [11352] WARNING: there is no transaction in progress
2026-09-29 01:03:15,699 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 01:03:15,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 01:03:15,953 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 01:03:15,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 01:03:16,195 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 01:03:16,196 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 01:03:16,440 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 01:03:16,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 01:03:16,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 01:03:16,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7zdl.c.gcp-upstream.internal';"
2026-09-29 01:03:17,167 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 01:03:17,168 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 01:03:17,419 [I] DB_PATCHES have already been applied
I20260929 01:03:17.425354 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskS2TZOM/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 01:03:17.425446 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260929 01:03:17.428834 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:45157
Sep 29, 2026 1:03:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 1:03:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 1:03:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 1:03:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 1:03:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 1:03:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 1:03:18 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 01:03:18.543411 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.361s user 0.005s sys 0.005s
Sep 29, 2026 1:03:37 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 1:03:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 1:03:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 1:03:37 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 1:03:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 1:03:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 1:03:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 1:03:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 1:03:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 1:03:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 1:03:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 1:03:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 1:03:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 1:03:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 1:03:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 1:03:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 1:03:37 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 1:03:38 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 01:03:39.227757 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 01:03:39 dist-test-slave-7zdl krb5kdc[11139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643819, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 01:03:39.328679 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 01:03:39.339170 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 01:03:40.176811 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 01:03:40.176932 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ranger-kms
I20260929 01:03:40.177035 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:60559
2026-09-29 01:03:40,357 [I] DB FLAVOR :POSTGRES
2026-09-29 01:03:40,357 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 01:03:40,357 [I] Checking connection
2026-09-29 01:03:40,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60559/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 01:03:40,602 [I] connection success
2026-09-29 01:03:40,602 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 01:03:40,602 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 01:03:40,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60559/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 01:03:40,838 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 01:03:40,838 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 01:03:40,839 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 01:03:40,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60559/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 01:03:41,107 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 01:03:41.557607 392 mini_ranger_kms.cc:310] Created kms service
I20260929 01:03:41.755208 392 mini_ranger_kms.cc:326] Created kudu user
I20260929 01:03:41.796574 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 01:03:41 dist-test-slave-7zdl krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643819, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 01:03:42.154114 392 mini_ranger_kms.cc:384] Added ranger policy
I20260929 01:03:42.154269 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 01:03:42.154291 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260929 01:03:42.157814 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11532
I20260929 01:03:42.157953 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:46345
Sep 29, 2026 1:03:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 1:03:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 1:03:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 1:03:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 1:03:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 1:03:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 01:03:44.451630 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.123s user 0.007s sys 0.158s
I20260929 01:03:44.451768 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 01:03:44.451881 392 mini_ranger_kms.cc:401] 127.0.98.20:46345/kms/v1/keys
Sep 29 01:03:44 dist-test-slave-7zdl krb5kdc[11139](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790643824, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 01:03:44 dist-test-slave-7zdl krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643824, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 01:03:45 dist-test-slave-7zdl krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643824, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 01:03:47 dist-test-slave-7zdl krb5kdc[11139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:03:47.565729 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42319
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42319
--ranger_config_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46345/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:03:47.693097 11583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:03:47.693344 11583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:03:47.693383 11583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:03:47.697324 11583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:03:47.697378 11583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:03:47.697399 11583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:03:47.697415 11583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:03:47.697430 11583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:03:47.702783 11583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:46345/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42319
--ranger_config_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:03:47.703820 11583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:03:47.705102 11583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:03:47.716564 11583 server_base.cc:1034] running on GCE node
W20260929 01:03:47.716784 11591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:47.716902 11588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:47.716784 11589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:47.717568 11583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:03:47.718708 11583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:03:47.719892 11583 hybrid_clock.cc:648] HybridClock initialized: now 1790643827719874 us; error 28 us; skew 500 ppm
Sep 29 01:03:47 dist-test-slave-7zdl krb5kdc[11139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:03:47.726287 11583 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:03:47.727185 11583 webserver.cc:460] Webserver started at http://127.0.98.62:33017/ using document root <none> and password file <none>
I20260929 01:03:47.727679 11583 fs_manager.cc:359] Metadata directory not provided
I20260929 01:03:47.727728 11583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:03:47.727887 11583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 01:03:47 dist-test-slave-7zdl krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 01:03:47.780373 11583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "8a12c0645a2144aab2f835a8ee2b900d"
format_stamp: "Formatted at 2026-09-29 01:03:47 on dist-test-slave-7zdl"
server_key: "a557e734ac39fbb8060e7d286921eba8"
server_key_iv: "aefe3ca643b6220f7fd8760f3f9cf942"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:47.781104 11583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "8a12c0645a2144aab2f835a8ee2b900d"
format_stamp: "Formatted at 2026-09-29 01:03:47 on dist-test-slave-7zdl"
server_key: "a557e734ac39fbb8060e7d286921eba8"
server_key_iv: "aefe3ca643b6220f7fd8760f3f9cf942"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:47.785981 11583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 01:03:47.800099 11598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:47.801499 11583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:03:47.801636 11583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "8a12c0645a2144aab2f835a8ee2b900d"
format_stamp: "Formatted at 2026-09-29 01:03:47 on dist-test-slave-7zdl"
server_key: "a557e734ac39fbb8060e7d286921eba8"
server_key_iv: "aefe3ca643b6220f7fd8760f3f9cf942"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:47.801777 11583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:03:47.821482 11583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:03:47.824388 11583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:03:47.824597 11583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:03:47.832450 11583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42319
I20260929 01:03:47.832495 11650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42319 every 8 connection(s)
I20260929 01:03:47.833387 11583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:03:47.836081 11651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:03:47.841305 11651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a12c0645a2144aab2f835a8ee2b900d: Bootstrap starting.
I20260929 01:03:47.843032 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 11583
I20260929 01:03:47.843151 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:03:47.843447 392 external_mini_cluster.cc:1428] Setting key 8f7dcd1e8613d1922c245702430bc182
I20260929 01:03:47.844173 11651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a12c0645a2144aab2f835a8ee2b900d: Neither blocks nor log segments found. Creating new log.
I20260929 01:03:47.845090 11651 log.cc:824] T 00000000000000000000000000000000 P 8a12c0645a2144aab2f835a8ee2b900d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:03:47.847028 11651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a12c0645a2144aab2f835a8ee2b900d: No bootstrap required, opened a new log
I20260929 01:03:47.851432 11651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a12c0645a2144aab2f835a8ee2b900d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a12c0645a2144aab2f835a8ee2b900d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42319 } }
I20260929 01:03:47.851665 11651 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a12c0645a2144aab2f835a8ee2b900d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:03:47.851737 11651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a12c0645a2144aab2f835a8ee2b900d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a12c0645a2144aab2f835a8ee2b900d, State: Initialized, Role: FOLLOWER
I20260929 01:03:47.852398 11651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a12c0645a2144aab2f835a8ee2b900d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a12c0645a2144aab2f835a8ee2b900d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42319 } }
I20260929 01:03:47.852650 11651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a12c0645a2144aab2f835a8ee2b900d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:03:47.852753 11651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a12c0645a2144aab2f835a8ee2b900d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:03:47.852859 11651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a12c0645a2144aab2f835a8ee2b900d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:03:47.853798 11651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a12c0645a2144aab2f835a8ee2b900d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a12c0645a2144aab2f835a8ee2b900d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42319 } }
Sep 29 01:03:47 dist-test-slave-7zdl krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:03:47.854177 11651 leader_election.cc:302] T 00000000000000000000000000000000 P 8a12c0645a2144aab2f835a8ee2b900d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a12c0645a2144aab2f835a8ee2b900d; no voters:
I20260929 01:03:47.854514 11651 leader_election.cc:288] T 00000000000000000000000000000000 P 8a12c0645a2144aab2f835a8ee2b900d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:03:47.854614 11656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a12c0645a2144aab2f835a8ee2b900d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:03:47.854959 11656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a12c0645a2144aab2f835a8ee2b900d [term 1 LEADER]: Becoming Leader. State: Replica: 8a12c0645a2144aab2f835a8ee2b900d, State: Running, Role: LEADER
I20260929 01:03:47.855299 11656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a12c0645a2144aab2f835a8ee2b900d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a12c0645a2144aab2f835a8ee2b900d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42319 } }
I20260929 01:03:47.856071 11651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a12c0645a2144aab2f835a8ee2b900d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:03:47.856489 11656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a12c0645a2144aab2f835a8ee2b900d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a12c0645a2144aab2f835a8ee2b900d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a12c0645a2144aab2f835a8ee2b900d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42319 } } }
I20260929 01:03:47.856658 11656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a12c0645a2144aab2f835a8ee2b900d [sys.catalog]: This master's current role is: LEADER
I20260929 01:03:47.856921 11656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a12c0645a2144aab2f835a8ee2b900d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a12c0645a2144aab2f835a8ee2b900d. Latest consensus state: current_term: 1 leader_uuid: "8a12c0645a2144aab2f835a8ee2b900d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a12c0645a2144aab2f835a8ee2b900d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42319 } } }
I20260929 01:03:47.857059 11656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a12c0645a2144aab2f835a8ee2b900d [sys.catalog]: This master's current role is: LEADER
I20260929 01:03:47.857363 11661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:03:47.860608 11661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:03:47.860692 11654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:47.844645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44950 (local address 127.0.98.62:42319)
0929 01:03:47.845109 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:47.845128 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:47.845170 (+ 42us) server_negotiation.cc:407] Connection header received
0929 01:03:47.846012 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:47.846036 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:47.846463 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:47.846848 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:47.848162 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:47.849662 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:47.850922 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:47.851614 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:47.852169 (+ 555us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:47.854864 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:47.854872 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:47.854893 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:03:47.854924 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:47.856949 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:47.857503 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:47.857509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:47.857515 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:47.857645 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:47.857995 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:47.858000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:47.858002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:47.858549 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:47.858722 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:47.859472 (+ 750us) server_negotiation.cc:299] Negotiation successful
0929 01:03:47.859746 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":137,"threads_started":1}
I20260929 01:03:47.861436 11651 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 01:03:47.869477 11661 catalog_manager.cc:1347] Generated new cluster ID: c3a5ae3a60204ca7aab42dab31690bb1
I20260929 01:03:47.869540 11661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:03:47.886142 11661 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:03:47.887573 11661 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:03:47.897295 11661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a12c0645a2144aab2f835a8ee2b900d: Generated new TSK 0
I20260929 01:03:47.898159 11661 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 01:03:48.621800 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42319
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:03:48.765616 11696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:03:48.765913 11696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:03:48.765988 11696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:03:48.771831 11696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:03:48.771911 11696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:03:48.771998 11696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:03:48.779018 11696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:46345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:03:48.780172 11696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:03:48.781188 11696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:03:48.790547 11704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:48.790766 11696 server_base.cc:1034] running on GCE node
W20260929 01:03:48.790542 11706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:48.791131 11703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:48.791527 11696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:03:48.792128 11696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:03:48.793310 11696 hybrid_clock.cc:648] HybridClock initialized: now 1790643828793280 us; error 42 us; skew 500 ppm
Sep 29 01:03:48 dist-test-slave-7zdl krb5kdc[11139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:03:48.796051 11696 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:03:48.796988 11696 webserver.cc:460] Webserver started at http://127.0.98.1:45457/ using document root <none> and password file <none>
I20260929 01:03:48.797541 11696 fs_manager.cc:359] Metadata directory not provided
I20260929 01:03:48.797637 11696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:03:48.797873 11696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 01:03:48 dist-test-slave-7zdl krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 01:03:48.820127 11696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "ad00523ec924434faa0f9f48419bf01f"
format_stamp: "Formatted at 2026-09-29 01:03:48 on dist-test-slave-7zdl"
server_key: "d4a6cfc7da553c050e86f676c8637102"
server_key_iv: "3e5b7fa93dba20f4151291730a0941d2"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:48.820842 11696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ad00523ec924434faa0f9f48419bf01f"
format_stamp: "Formatted at 2026-09-29 01:03:48 on dist-test-slave-7zdl"
server_key: "d4a6cfc7da553c050e86f676c8637102"
server_key_iv: "3e5b7fa93dba20f4151291730a0941d2"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:48.832343 11696 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.005s sys 0.000s
I20260929 01:03:48.859169 11714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:48.861027 11696 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260929 01:03:48.861164 11696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "ad00523ec924434faa0f9f48419bf01f"
format_stamp: "Formatted at 2026-09-29 01:03:48 on dist-test-slave-7zdl"
server_key: "d4a6cfc7da553c050e86f676c8637102"
server_key_iv: "3e5b7fa93dba20f4151291730a0941d2"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:48.861311 11696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:03:48.882746 11696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:03:48.886346 11696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:03:48.886565 11696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:03:48.887284 11696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:03:48.888214 11696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:03:48.888288 11696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:48.888378 11696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:03:48.888414 11696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:48.900019 11696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39477
I20260929 01:03:48.900027 11828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39477 every 8 connection(s)
I20260929 01:03:48.901103 11696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:03:48.910703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 11696
I20260929 01:03:48.910856 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:03:48.911201 392 external_mini_cluster.cc:1428] Setting key fe8ce5edf07f162f24acdc5ce2495b28
Sep 29 01:03:48 dist-test-slave-7zdl krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:03:48.918609 11831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:48.903236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42219 (local address 127.0.98.62:42319)
0929 01:03:48.903494 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:48.903498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:48.904380 (+ 882us) server_negotiation.cc:407] Connection header received
0929 01:03:48.905451 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:48.905456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:48.905515 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:48.905637 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:48.906869 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:48.907413 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:48.908565 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:48.908794 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:48.908905 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:48.912157 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:48.912160 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:48.912163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:03:48.912185 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:48.915810 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:48.916474 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:48.916481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:48.916485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:48.916583 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:48.917032 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:48.917038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:48.917042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:48.917307 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:48.917386 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:48.918176 (+ 790us) server_negotiation.cc:299] Negotiation successful
0929 01:03:48.918330 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":77,"threads_started":1}
I20260929 01:03:48.920692 11832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:48.903747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.1:42219)
0929 01:03:48.904227 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:48.904281 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:48.905147 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:48.905815 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:48.905822 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:48.906098 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:03:48.906702 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:48.906718 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:48.907545 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:48.907549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:48.908442 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:48.908450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:48.908950 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:48.908955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:48.909132 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:48.909855 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:03:48.909870 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:03:48.911982 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:03:48.915957 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:03:48.915965 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:03:48.915984 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:03:48.916345 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:03:48.916704 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:03:48.916710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:03:48.916715 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:03:48.916912 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:03:48.917483 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:03:48.917492 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:03:48.917854 (+ 362us) client_negotiation.cc:769] Sending connection context
0929 01:03:48.919135 (+ 1281us) client_negotiation.cc:240] Negotiation successful
0929 01:03:48.919540 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":148,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 01:03:48.922478 11829 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42319
I20260929 01:03:48.922744 11829 heartbeater.cc:464] Registering TS with master...
I20260929 01:03:48.923314 11829 heartbeater.cc:511] Master 127.0.98.62:42319 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 01:03:48.925141 11611 ts_manager.cc:194] Registered new tserver with Master: ad00523ec924434faa0f9f48419bf01f (127.0.98.1:39477)
I20260929 01:03:48.926604 11611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42219
Sep 29 01:03:48 dist-test-slave-7zdl krb5kdc[11139](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790643828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 01:03:48.955430 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42319
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:03:49.116482 11835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:03:49.116842 11835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:03:49.116910 11835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:03:49.122926 11835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:03:49.122996 11835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:03:49.123081 11835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:03:49.130761 11835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:46345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:03:49.132056 11835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:03:49.133294 11835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:03:49.144981 11842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:49.145179 11844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:49.145359 11841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:49.145526 11835 server_base.cc:1034] running on GCE node
I20260929 01:03:49.145928 11835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:03:49.146648 11835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:03:49.147832 11835 hybrid_clock.cc:648] HybridClock initialized: now 1790643829147764 us; error 82 us; skew 500 ppm
Sep 29 01:03:49 dist-test-slave-7zdl krb5kdc[11139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:03:49.151489 11835 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:03:49.152541 11835 webserver.cc:460] Webserver started at http://127.0.98.2:40027/ using document root <none> and password file <none>
I20260929 01:03:49.153224 11835 fs_manager.cc:359] Metadata directory not provided
I20260929 01:03:49.153322 11835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:03:49.153584 11835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 01:03:49 dist-test-slave-7zdl krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 01:03:49.176340 11835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "ce2fcf9c2da241a2a3bf16ce8d2e7d8b"
format_stamp: "Formatted at 2026-09-29 01:03:49 on dist-test-slave-7zdl"
server_key: "8aa07c118063eca49f39dee0f49a248c"
server_key_iv: "1542ffa6d381b27bdfcd394db4e92d71"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:49.177102 11835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ce2fcf9c2da241a2a3bf16ce8d2e7d8b"
format_stamp: "Formatted at 2026-09-29 01:03:49 on dist-test-slave-7zdl"
server_key: "8aa07c118063eca49f39dee0f49a248c"
server_key_iv: "1542ffa6d381b27bdfcd394db4e92d71"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:49.183251 11835 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 01:03:49.203380 11851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:49.204878 11835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:03:49.205092 11835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "ce2fcf9c2da241a2a3bf16ce8d2e7d8b"
format_stamp: "Formatted at 2026-09-29 01:03:49 on dist-test-slave-7zdl"
server_key: "8aa07c118063eca49f39dee0f49a248c"
server_key_iv: "1542ffa6d381b27bdfcd394db4e92d71"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:49.205251 11835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:03:49.240310 11835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:03:49.245843 11835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:03:49.246054 11835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:03:49.246779 11835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:03:49.247870 11835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:03:49.247937 11835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:49.247999 11835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:03:49.248046 11835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:49.260001 11835 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45979
I20260929 01:03:49.260059 11964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45979 every 8 connection(s)
I20260929 01:03:49.261188 11835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:03:49.271265 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 11835
I20260929 01:03:49.271417 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:03:49.271744 392 external_mini_cluster.cc:1428] Setting key a08a563baa49c68eb513f4cadeb00ea6
Sep 29 01:03:49 dist-test-slave-7zdl krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:03:49.280608 11831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:49.263315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39103 (local address 127.0.98.62:42319)
0929 01:03:49.263465 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:49.263470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:49.264437 (+ 967us) server_negotiation.cc:407] Connection header received
0929 01:03:49.265556 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:49.265560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:49.265617 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:49.265727 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:49.267229 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.268078 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.269796 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.270167 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.270292 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:49.276056 (+ 5764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:49.276060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:49.276063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:03:49.276094 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:49.277961 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:49.278629 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:49.278633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:49.278635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:49.278723 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:49.279182 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:49.279186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:49.279188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:49.279436 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:49.279509 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:49.280270 (+ 761us) server_negotiation.cc:299] Negotiation successful
0929 01:03:49.280404 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 01:03:49.282196 11967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:49.263749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.2:39103)
0929 01:03:49.264257 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:49.264308 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:49.265315 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:49.265924 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:49.265933 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:49.266315 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:03:49.267045 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:49.267067 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.268244 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.268248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:49.269655 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:49.269666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.270344 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.270351 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:49.270554 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:49.273979 (+ 3425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:03:49.274005 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:03:49.275847 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:03:49.278106 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:03:49.278112 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:03:49.278129 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:03:49.278509 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:03:49.278866 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:03:49.278869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:03:49.278871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:03:49.279064 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:03:49.279590 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:03:49.279596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:03:49.279961 (+ 365us) client_negotiation.cc:769] Sending connection context
0929 01:03:49.280774 (+ 813us) client_negotiation.cc:240] Negotiation successful
0929 01:03:49.281128 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":147,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 01:03:49.283619 11965 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42319
I20260929 01:03:49.283875 11965 heartbeater.cc:464] Registering TS with master...
I20260929 01:03:49.284478 11965 heartbeater.cc:511] Master 127.0.98.62:42319 requested a full tablet report, sending...
I20260929 01:03:49.285691 11611 ts_manager.cc:194] Registered new tserver with Master: ce2fcf9c2da241a2a3bf16ce8d2e7d8b (127.0.98.2:45979)
I20260929 01:03:49.286556 11611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 01:03:49.306849 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42319
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:03:49 dist-test-slave-7zdl krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260929 01:03:49.468691 11970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:03:49.469060 11970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:03:49.469134 11970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:03:49.475077 11970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:03:49.475152 11970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:03:49.475250 11970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:03:49.482944 11970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:46345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:03:49.484349 11970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:03:49.485669 11970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:03:49.498025 11975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:49.498132 11978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:03:49.497857 11976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:03:49.499298 11970 server_base.cc:1034] running on GCE node
I20260929 01:03:49.499754 11970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:03:49.500512 11970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:03:49.501710 11970 hybrid_clock.cc:648] HybridClock initialized: now 1790643829501677 us; error 49 us; skew 500 ppm
Sep 29 01:03:49 dist-test-slave-7zdl krb5kdc[11139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:03:49.505398 11970 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:03:49.506565 11970 webserver.cc:460] Webserver started at http://127.0.98.3:41217/ using document root <none> and password file <none>
I20260929 01:03:49.507349 11970 fs_manager.cc:359] Metadata directory not provided
I20260929 01:03:49.507440 11970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:03:49.507668 11970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 01:03:49 dist-test-slave-7zdl krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 01:03:49.561759 11970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "9164820a2887463eb2835b4dbb76b505"
format_stamp: "Formatted at 2026-09-29 01:03:49 on dist-test-slave-7zdl"
server_key: "b624479fe798fcdbdc19e0546c783ea0"
server_key_iv: "76c05e7303391ea7a51e68e06c2c0e6d"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:49.562475 11970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9164820a2887463eb2835b4dbb76b505"
format_stamp: "Formatted at 2026-09-29 01:03:49 on dist-test-slave-7zdl"
server_key: "b624479fe798fcdbdc19e0546c783ea0"
server_key_iv: "76c05e7303391ea7a51e68e06c2c0e6d"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:49.572624 11970 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.007s sys 0.000s
I20260929 01:03:49.618718 11985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:49.620838 11970 fs_manager.cc:725] Time spent opening block manager: real 0.024s user 0.003s sys 0.000s
I20260929 01:03:49.621001 11970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "9164820a2887463eb2835b4dbb76b505"
format_stamp: "Formatted at 2026-09-29 01:03:49 on dist-test-slave-7zdl"
server_key: "b624479fe798fcdbdc19e0546c783ea0"
server_key_iv: "76c05e7303391ea7a51e68e06c2c0e6d"
server_key_version: "kuduclusterkey@0"
I20260929 01:03:49.621176 11970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:03:49.672920 11970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:03:49.676816 11970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:03:49.677075 11970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:03:49.677842 11970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:03:49.680401 11970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:03:49.680526 11970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:49.680630 11970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:03:49.680696 11970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:03:49.707798 11970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45281
I20260929 01:03:49.709102 11970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:03:49.712419 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 11970
I20260929 01:03:49.712553 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:03:49.712837 392 external_mini_cluster.cc:1428] Setting key 9c0e6db5cdb2d6f1f633ca7e4652148a
I20260929 01:03:49.723523 12099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45281 every 8 connection(s)
Sep 29 01:03:49 dist-test-slave-7zdl krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:03:49.756004 11831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:49.727800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44609 (local address 127.0.98.62:42319)
0929 01:03:49.728080 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:49.728084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:49.741013 (+ 12929us) server_negotiation.cc:407] Connection header received
0929 01:03:49.742403 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:49.742406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:49.742462 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:49.742514 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:49.744331 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.745118 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.746916 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.747247 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.747328 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:49.750292 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:49.750297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:49.750300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:03:49.750327 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:49.753598 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:49.754232 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:49.754236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:49.754238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:49.754318 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:49.754700 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:49.754703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:49.754705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:49.754950 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:49.755034 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:49.755697 (+ 663us) server_negotiation.cc:299] Negotiation successful
0929 01:03:49.755820 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
I20260929 01:03:49.757828 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:49.726130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42319 (local address 127.0.98.3:44609)
0929 01:03:49.739170 (+ 13040us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:49.739252 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:49.742125 (+ 2873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:49.743020 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:49.743032 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:49.743418 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:03:49.744163 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:49.744182 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.745283 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.745289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:49.746738 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:49.746759 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.747545 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.747549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:49.747748 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:49.748408 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:03:49.748429 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:03:49.750097 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:03:49.753734 (+ 3637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:03:49.753740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:03:49.753757 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:03:49.754118 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:03:49.754416 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:03:49.754418 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:03:49.754420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:03:49.754591 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:03:49.755099 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:03:49.755106 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:03:49.755418 (+ 312us) client_negotiation.cc:769] Sending connection context
0929 01:03:49.756235 (+ 817us) client_negotiation.cc:240] Negotiation successful
0929 01:03:49.756597 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12751,"thread_start_us":147,"threads_started":1}
I20260929 01:03:49.759106 12100 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42319
I20260929 01:03:49.759351 12100 heartbeater.cc:464] Registering TS with master...
I20260929 01:03:49.759927 12100 heartbeater.cc:511] Master 127.0.98.62:42319 requested a full tablet report, sending...
I20260929 01:03:49.761004 11611 ts_manager.cc:194] Registered new tserver with Master: 9164820a2887463eb2835b4dbb76b505 (127.0.98.3:45281)
I20260929 01:03:49.761744 11611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44609
Sep 29 01:03:49 dist-test-slave-7zdl krb5kdc[11139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 01:03:49.765532 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:03:49.777477 11831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:49.767838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44958 (local address 127.0.98.62:42319)
0929 01:03:49.767962 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:49.767966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:49.768556 (+ 590us) server_negotiation.cc:407] Connection header received
0929 01:03:49.768744 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:49.768747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:49.768791 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:49.768841 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:03:49.769804 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.770641 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.771850 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.772168 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.772240 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:49.774003 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:03:49.774007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:03:49.774010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:03:49.774034 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:03:49.775746 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:49.776211 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:49.776214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:49.776216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:49.776292 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:03:49.776580 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:03:49.776583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:03:49.776584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:03:49.776774 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:03:49.776869 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:49.777194 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 01:03:49.777302 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 01:03:49.782305 11611 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:03:49.787343 11611 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:03:49.822002 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:49.800107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45281 (local address 127.0.0.1:46894)
0929 01:03:49.804467 (+ 4360us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:49.804520 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:49.804643 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:49.805834 (+ 1191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:49.805843 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:49.805866 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:03:49.806020 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:49.806026 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.808686 (+ 2660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.808690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:49.819961 (+ 11271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:49.819975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.821442 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.821445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:49.821564 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:49.821607 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 01:03:49.821718 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 01:03:49.821840 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3400,"thread_start_us":95,"threads_started":1}
I20260929 01:03:49.822901 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:49.801043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46894 (local address 127.0.98.3:45281)
0929 01:03:49.805200 (+ 4157us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:49.805212 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:49.805235 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:03:49.805358 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:49.805364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:49.805631 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:49.806178 (+ 547us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:03:49.806312 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.808550 (+ 2238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.820154 (+ 11604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.821081 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.821190 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:49.821369 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:49.822622 (+ 1253us) server_negotiation.cc:299] Negotiation successful
0929 01:03:49.822730 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4046,"thread_start_us":103,"threads_started":1}
I20260929 01:03:49.826236 12033 tablet_service.cc:1467] Processing CreateTablet for tablet 70f661b2e85c41638ab202aba9f28bb4 (DEFAULT_TABLE table=test-table [id=8598dc2d94784464bf3f06a59f90c844]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:03:49.827333 12033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70f661b2e85c41638ab202aba9f28bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:03:49.830566 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:49.803473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45979 (local address 127.0.0.1:36724)
0929 01:03:49.821372 (+ 17899us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:49.821386 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:49.821467 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:49.823887 (+ 2420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:49.823890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:49.823909 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:03:49.824008 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:49.824014 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.825489 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.825493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:49.826668 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:49.826678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.830173 (+ 3495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.830177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:49.830278 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:49.830302 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 01:03:49.830363 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 01:03:49.830432 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17833,"thread_start_us":85,"threads_started":1}
I20260929 01:03:49.832888 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:49.803868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36724 (local address 127.0.98.2:45979)
0929 01:03:49.821701 (+ 17833us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:49.821707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:49.821730 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:03:49.821801 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:49.821807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:49.822010 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:49.822150 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:03:49.824228 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.825356 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.828431 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.829750 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.829853 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:49.832610 (+ 2757us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:49.832664 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 01:03:49.832757 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17725,"thread_start_us":114,"threads_started":1}
I20260929 01:03:49.835743 11881 tablet_service.cc:1467] Processing CreateTablet for tablet 70f661b2e85c41638ab202aba9f28bb4 (DEFAULT_TABLE table=test-table [id=8598dc2d94784464bf3f06a59f90c844]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:03:49.836771 11881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70f661b2e85c41638ab202aba9f28bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:03:49.843427 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:49.802490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39477 (local address 127.0.0.1:34554)
0929 01:03:49.804859 (+ 2369us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:49.804870 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:49.804951 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:49.821095 (+ 16144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:49.821099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:49.821124 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:03:49.821229 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:49.821235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.840663 (+ 19428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.840666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:49.841898 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:49.841908 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.843048 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.843052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:49.843145 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:49.843171 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 01:03:49.843218 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 01:03:49.843294 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2312,"thread_start_us":94,"threads_started":1}
I20260929 01:03:49.844165 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:49.802939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34554 (local address 127.0.98.1:39477)
0929 01:03:49.820572 (+ 17633us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:49.820580 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:49.820607 (+ 27us) server_negotiation.cc:407] Connection header received
0929 01:03:49.820693 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:49.820699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:49.820957 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:49.821125 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:03:49.822558 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.836619 (+ 14061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.842071 (+ 5452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.842915 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.843723 (+ 808us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:49.843865 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:49.843930 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 01:03:49.844025 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17503,"thread_start_us":90,"threads_started":1}
I20260929 01:03:49.844673 12119 tablet_bootstrap.cc:492] T 70f661b2e85c41638ab202aba9f28bb4 P ce2fcf9c2da241a2a3bf16ce8d2e7d8b: Bootstrap starting.
I20260929 01:03:49.847035 11747 tablet_service.cc:1467] Processing CreateTablet for tablet 70f661b2e85c41638ab202aba9f28bb4 (DEFAULT_TABLE table=test-table [id=8598dc2d94784464bf3f06a59f90c844]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:03:49.847939 12119 tablet_bootstrap.cc:654] T 70f661b2e85c41638ab202aba9f28bb4 P ce2fcf9c2da241a2a3bf16ce8d2e7d8b: Neither blocks nor log segments found. Creating new log.
I20260929 01:03:49.848094 11747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70f661b2e85c41638ab202aba9f28bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:03:49.848734 12119 log.cc:824] T 70f661b2e85c41638ab202aba9f28bb4 P ce2fcf9c2da241a2a3bf16ce8d2e7d8b: Log is configured to *not* fsync() on all Append() calls
I20260929 01:03:49.850749 12119 tablet_bootstrap.cc:492] T 70f661b2e85c41638ab202aba9f28bb4 P ce2fcf9c2da241a2a3bf16ce8d2e7d8b: No bootstrap required, opened a new log
I20260929 01:03:49.850978 12119 ts_tablet_manager.cc:1397] T 70f661b2e85c41638ab202aba9f28bb4 P ce2fcf9c2da241a2a3bf16ce8d2e7d8b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260929 01:03:49.854786 12121 tablet_bootstrap.cc:492] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f: Bootstrap starting.
I20260929 01:03:49.856048 12118 tablet_bootstrap.cc:492] T 70f661b2e85c41638ab202aba9f28bb4 P 9164820a2887463eb2835b4dbb76b505: Bootstrap starting.
I20260929 01:03:49.856473 12119 raft_consensus.cc:348] T 70f661b2e85c41638ab202aba9f28bb4 P ce2fcf9c2da241a2a3bf16ce8d2e7d8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9164820a2887463eb2835b4dbb76b505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45281 } } peers { permanent_uuid: "ad00523ec924434faa0f9f48419bf01f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39477 } } peers { permanent_uuid: "ce2fcf9c2da241a2a3bf16ce8d2e7d8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45979 } }
I20260929 01:03:49.856737 12119 raft_consensus.cc:374] T 70f661b2e85c41638ab202aba9f28bb4 P ce2fcf9c2da241a2a3bf16ce8d2e7d8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:03:49.856796 12119 raft_consensus.cc:729] T 70f661b2e85c41638ab202aba9f28bb4 P ce2fcf9c2da241a2a3bf16ce8d2e7d8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce2fcf9c2da241a2a3bf16ce8d2e7d8b, State: Initialized, Role: FOLLOWER
I20260929 01:03:49.857434 12119 consensus_queue.cc:260] T 70f661b2e85c41638ab202aba9f28bb4 P ce2fcf9c2da241a2a3bf16ce8d2e7d8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9164820a2887463eb2835b4dbb76b505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45281 } } peers { permanent_uuid: "ad00523ec924434faa0f9f48419bf01f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39477 } } peers { permanent_uuid: "ce2fcf9c2da241a2a3bf16ce8d2e7d8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45979 } }
I20260929 01:03:49.858969 12118 tablet_bootstrap.cc:654] T 70f661b2e85c41638ab202aba9f28bb4 P 9164820a2887463eb2835b4dbb76b505: Neither blocks nor log segments found. Creating new log.
I20260929 01:03:49.859438 12119 ts_tablet_manager.cc:1428] T 70f661b2e85c41638ab202aba9f28bb4 P ce2fcf9c2da241a2a3bf16ce8d2e7d8b: Time spent starting tablet: real 0.008s user 0.003s sys 0.002s
I20260929 01:03:49.859776 11965 heartbeater.cc:503] Master 127.0.98.62:42319 was elected leader, sending a full tablet report...
I20260929 01:03:49.859824 12118 log.cc:824] T 70f661b2e85c41638ab202aba9f28bb4 P 9164820a2887463eb2835b4dbb76b505: Log is configured to *not* fsync() on all Append() calls
I20260929 01:03:49.860180 12121 tablet_bootstrap.cc:654] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f: Neither blocks nor log segments found. Creating new log.
I20260929 01:03:49.861003 12121 log.cc:824] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f: Log is configured to *not* fsync() on all Append() calls
I20260929 01:03:49.863524 12121 tablet_bootstrap.cc:492] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f: No bootstrap required, opened a new log
I20260929 01:03:49.863792 12121 ts_tablet_manager.cc:1397] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260929 01:03:49.872572 12118 tablet_bootstrap.cc:492] T 70f661b2e85c41638ab202aba9f28bb4 P 9164820a2887463eb2835b4dbb76b505: No bootstrap required, opened a new log
I20260929 01:03:49.872527 12121 raft_consensus.cc:348] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9164820a2887463eb2835b4dbb76b505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45281 } } peers { permanent_uuid: "ad00523ec924434faa0f9f48419bf01f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39477 } } peers { permanent_uuid: "ce2fcf9c2da241a2a3bf16ce8d2e7d8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45979 } }
I20260929 01:03:49.872784 12118 ts_tablet_manager.cc:1397] T 70f661b2e85c41638ab202aba9f28bb4 P 9164820a2887463eb2835b4dbb76b505: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20260929 01:03:49.872825 12121 raft_consensus.cc:374] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:03:49.872911 12121 raft_consensus.cc:729] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad00523ec924434faa0f9f48419bf01f, State: Initialized, Role: FOLLOWER
I20260929 01:03:49.873440 12121 consensus_queue.cc:260] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9164820a2887463eb2835b4dbb76b505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45281 } } peers { permanent_uuid: "ad00523ec924434faa0f9f48419bf01f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39477 } } peers { permanent_uuid: "ce2fcf9c2da241a2a3bf16ce8d2e7d8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45979 } }
I20260929 01:03:49.874564 12121 ts_tablet_manager.cc:1428] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f: Time spent starting tablet: real 0.011s user 0.004s sys 0.002s
I20260929 01:03:49.875005 11829 heartbeater.cc:503] Master 127.0.98.62:42319 was elected leader, sending a full tablet report...
I20260929 01:03:49.877081 12118 raft_consensus.cc:348] T 70f661b2e85c41638ab202aba9f28bb4 P 9164820a2887463eb2835b4dbb76b505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9164820a2887463eb2835b4dbb76b505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45281 } } peers { permanent_uuid: "ad00523ec924434faa0f9f48419bf01f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39477 } } peers { permanent_uuid: "ce2fcf9c2da241a2a3bf16ce8d2e7d8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45979 } }
I20260929 01:03:49.877357 12118 raft_consensus.cc:374] T 70f661b2e85c41638ab202aba9f28bb4 P 9164820a2887463eb2835b4dbb76b505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:03:49.877418 12118 raft_consensus.cc:729] T 70f661b2e85c41638ab202aba9f28bb4 P 9164820a2887463eb2835b4dbb76b505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9164820a2887463eb2835b4dbb76b505, State: Initialized, Role: FOLLOWER
I20260929 01:03:49.877876 12118 consensus_queue.cc:260] T 70f661b2e85c41638ab202aba9f28bb4 P 9164820a2887463eb2835b4dbb76b505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9164820a2887463eb2835b4dbb76b505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45281 } } peers { permanent_uuid: "ad00523ec924434faa0f9f48419bf01f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39477 } } peers { permanent_uuid: "ce2fcf9c2da241a2a3bf16ce8d2e7d8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45979 } }
I20260929 01:03:49.878893 12100 heartbeater.cc:503] Master 127.0.98.62:42319 was elected leader, sending a full tablet report...
I20260929 01:03:49.878953 12118 ts_tablet_manager.cc:1428] T 70f661b2e85c41638ab202aba9f28bb4 P 9164820a2887463eb2835b4dbb76b505: Time spent starting tablet: real 0.006s user 0.001s sys 0.006s
I20260929 01:03:49.880580 12125 raft_consensus.cc:482] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:03:49.880724 12125 raft_consensus.cc:504] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9164820a2887463eb2835b4dbb76b505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45281 } } peers { permanent_uuid: "ad00523ec924434faa0f9f48419bf01f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39477 } } peers { permanent_uuid: "ce2fcf9c2da241a2a3bf16ce8d2e7d8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45979 } }
I20260929 01:03:49.882894 12125 leader_election.cc:288] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9164820a2887463eb2835b4dbb76b505 (127.0.98.3:45281), ce2fcf9c2da241a2a3bf16ce8d2e7d8b (127.0.98.2:45979)
I20260929 01:03:49.887962 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:49.882966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41355 (local address 127.0.98.2:45979)
0929 01:03:49.883091 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:49.883097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:49.883367 (+ 270us) server_negotiation.cc:407] Connection header received
0929 01:03:49.883591 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:49.883615 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:49.883674 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:49.883729 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:03:49.884190 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.885004 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.886590 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.887242 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.887328 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:49.887419 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:49.887736 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 01:03:49.887813 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 01:03:49.888423 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:49.882543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45979 (local address 127.0.98.1:41355)
0929 01:03:49.883277 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:49.883293 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:49.883447 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:49.883872 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:49.883878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:49.883893 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:03:49.884059 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:49.884069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.885146 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.885149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:49.886451 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:49.886461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.887504 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.887507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:49.887606 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:49.887628 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 01:03:49.888186 (+ 558us) client_negotiation.cc:240] Negotiation successful
0929 01:03:49.888268 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":109,"threads_started":1}
I20260929 01:03:49.889230 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:49.882681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54633 (local address 127.0.98.3:45281)
0929 01:03:49.882825 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:49.882831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:49.883735 (+ 904us) server_negotiation.cc:407] Connection header received
0929 01:03:49.883902 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:49.883907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:49.883954 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:49.884004 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:03:49.884368 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.885078 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.887425 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.887993 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.888072 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:49.888130 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:49.889038 (+ 908us) server_negotiation.cc:299] Negotiation successful
0929 01:03:49.889116 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 01:03:49.889173 11901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f661b2e85c41638ab202aba9f28bb4" candidate_uuid: "ad00523ec924434faa0f9f48419bf01f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce2fcf9c2da241a2a3bf16ce8d2e7d8b" is_pre_election: true
I20260929 01:03:49.889510 11901 raft_consensus.cc:2454] T 70f661b2e85c41638ab202aba9f28bb4 P ce2fcf9c2da241a2a3bf16ce8d2e7d8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad00523ec924434faa0f9f48419bf01f in term 0.
I20260929 01:03:49.890059 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:49.882543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45281 (local address 127.0.98.1:54633)
0929 01:03:49.883660 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:49.883673 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:03:49.883791 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:03:49.884125 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:03:49.884128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:03:49.884146 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:03:49.884258 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:49.884264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.885715 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.885718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:49.887288 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:03:49.887298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.888787 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.888790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:03:49.888897 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:49.888919 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 01:03:49.889401 (+ 482us) client_negotiation.cc:240] Negotiation successful
0929 01:03:49.889530 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1026,"mutex_wait_us":472,"thread_start_us":93,"threads_started":1}
I20260929 01:03:49.890107 11717 leader_election.cc:302] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad00523ec924434faa0f9f48419bf01f, ce2fcf9c2da241a2a3bf16ce8d2e7d8b; no voters:
I20260929 01:03:49.890730 12053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f661b2e85c41638ab202aba9f28bb4" candidate_uuid: "ad00523ec924434faa0f9f48419bf01f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9164820a2887463eb2835b4dbb76b505" is_pre_election: true
I20260929 01:03:49.890976 12125 raft_consensus.cc:2781] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:03:49.891098 12125 raft_consensus.cc:482] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:03:49.891064 12053 raft_consensus.cc:2454] T 70f661b2e85c41638ab202aba9f28bb4 P 9164820a2887463eb2835b4dbb76b505 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad00523ec924434faa0f9f48419bf01f in term 0.
I20260929 01:03:49.891161 12125 raft_consensus.cc:3037] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:03:49.892385 12125 raft_consensus.cc:504] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9164820a2887463eb2835b4dbb76b505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45281 } } peers { permanent_uuid: "ad00523ec924434faa0f9f48419bf01f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39477 } } peers { permanent_uuid: "ce2fcf9c2da241a2a3bf16ce8d2e7d8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45979 } }
I20260929 01:03:49.892796 12125 leader_election.cc:288] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f [CANDIDATE]: Term 1 election: Requested vote from peers 9164820a2887463eb2835b4dbb76b505 (127.0.98.3:45281), ce2fcf9c2da241a2a3bf16ce8d2e7d8b (127.0.98.2:45979)
I20260929 01:03:49.893204 12053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f661b2e85c41638ab202aba9f28bb4" candidate_uuid: "ad00523ec924434faa0f9f48419bf01f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9164820a2887463eb2835b4dbb76b505"
I20260929 01:03:49.893229 11901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70f661b2e85c41638ab202aba9f28bb4" candidate_uuid: "ad00523ec924434faa0f9f48419bf01f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce2fcf9c2da241a2a3bf16ce8d2e7d8b"
I20260929 01:03:49.893328 12053 raft_consensus.cc:3037] T 70f661b2e85c41638ab202aba9f28bb4 P 9164820a2887463eb2835b4dbb76b505 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:03:49.893353 11901 raft_consensus.cc:3037] T 70f661b2e85c41638ab202aba9f28bb4 P ce2fcf9c2da241a2a3bf16ce8d2e7d8b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:03:49.894629 12053 raft_consensus.cc:2454] T 70f661b2e85c41638ab202aba9f28bb4 P 9164820a2887463eb2835b4dbb76b505 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad00523ec924434faa0f9f48419bf01f in term 1.
I20260929 01:03:49.894694 11901 raft_consensus.cc:2454] T 70f661b2e85c41638ab202aba9f28bb4 P ce2fcf9c2da241a2a3bf16ce8d2e7d8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad00523ec924434faa0f9f48419bf01f in term 1.
I20260929 01:03:49.895116 11719 leader_election.cc:302] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9164820a2887463eb2835b4dbb76b505, ad00523ec924434faa0f9f48419bf01f; no voters:
I20260929 01:03:49.895331 12125 raft_consensus.cc:2781] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:03:49.895556 12125 raft_consensus.cc:686] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f [term 1 LEADER]: Becoming Leader. State: Replica: ad00523ec924434faa0f9f48419bf01f, State: Running, Role: LEADER
I20260929 01:03:49.896049 12125 consensus_queue.cc:237] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9164820a2887463eb2835b4dbb76b505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45281 } } peers { permanent_uuid: "ad00523ec924434faa0f9f48419bf01f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39477 } } peers { permanent_uuid: "ce2fcf9c2da241a2a3bf16ce8d2e7d8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45979 } }
I20260929 01:03:49.899979 11615 catalog_manager.cc:5495] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f reported cstate change: term changed from 0 to 1, leader changed from <none> to ad00523ec924434faa0f9f48419bf01f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ad00523ec924434faa0f9f48419bf01f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9164820a2887463eb2835b4dbb76b505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad00523ec924434faa0f9f48419bf01f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce2fcf9c2da241a2a3bf16ce8d2e7d8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45979 } health_report { overall_health: UNKNOWN } } }
W20260929 01:03:49.903293 11830 tablet.cc:2369] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:03:49.936890 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:49.931694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34560 (local address 127.0.98.1:39477)
0929 01:03:49.931820 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 01:03:49.931827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:03:49.931843 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:03:49.932059 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:03:49.932063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:03:49.932142 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:03:49.932254 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:03:49.932675 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.933496 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.934737 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:03:49.935107 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:03:49.935205 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:03:49.935644 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:03:49.936205 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:03:49.936322 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 01:03:49.936659 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 01:03:49.936719 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 01:03:49.945858 11901 raft_consensus.cc:1261] T 70f661b2e85c41638ab202aba9f28bb4 P ce2fcf9c2da241a2a3bf16ce8d2e7d8b [term 1 FOLLOWER]: Refusing update from remote peer ad00523ec924434faa0f9f48419bf01f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:03:49.947045 12053 raft_consensus.cc:1261] T 70f661b2e85c41638ab202aba9f28bb4 P 9164820a2887463eb2835b4dbb76b505 [term 1 FOLLOWER]: Refusing update from remote peer ad00523ec924434faa0f9f48419bf01f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:03:49.947505 12125 consensus_queue.cc:1035] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce2fcf9c2da241a2a3bf16ce8d2e7d8b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:03:49.947686 12130 consensus_queue.cc:1035] T 70f661b2e85c41638ab202aba9f28bb4 P ad00523ec924434faa0f9f48419bf01f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9164820a2887463eb2835b4dbb76b505" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:03:49.960690 12133 mvcc.cc:205] Tried to move back new op lower bound from 7334477127450574848 to 7334477127257374720. Current Snapshot: MvccSnapshot[applied={T|T < 7334477127450574848}]
I20260929 01:03:49.965454 12134 mvcc.cc:205] Tried to move back new op lower bound from 7334477127450574848 to 7334477127257374720. Current Snapshot: MvccSnapshot[applied={T|T < 7334477127450574848}]
I20260929 01:03:49.979393 12136 mvcc.cc:205] Tried to move back new op lower bound from 7334477127450574848 to 7334477127257374720. Current Snapshot: MvccSnapshot[applied={T|T < 7334477127450574848}]
I20260929 01:03:49.981976 11611 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:03:49.982164 11611 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:03:49.985325 11611 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a12c0645a2144aab2f835a8ee2b900d: Sending DeleteTablet for 3 replicas of tablet 70f661b2e85c41638ab202aba9f28bb4
I20260929 01:03:49.986164 12033 tablet_service.cc:1514] Processing DeleteTablet for tablet 70f661b2e85c41638ab202aba9f28bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:03:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46894
I20260929 01:03:49.986275 11881 tablet_service.cc:1514] Processing DeleteTablet for tablet 70f661b2e85c41638ab202aba9f28bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:03:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36724
I20260929 01:03:49.987095 11747 tablet_service.cc:1514] Processing DeleteTablet for tablet 70f661b2e85c41638ab202aba9f28bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:03:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34554
I20260929 01:03:49.987641 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 11696
I20260929 01:03:49.990381 12143 tablet_replica.cc:330] T 70f661b2e85c41638ab202aba9f28bb4 P ce2fcf9c2da241a2a3bf16ce8d2e7d8b: stopping tablet replica
I20260929 01:03:49.990736 12143 raft_consensus.cc:2229] T 70f661b2e85c41638ab202aba9f28bb4 P ce2fcf9c2da241a2a3bf16ce8d2e7d8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:03:49.991134 12143 raft_consensus.cc:2258] T 70f661b2e85c41638ab202aba9f28bb4 P ce2fcf9c2da241a2a3bf16ce8d2e7d8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:03:49.993996 12142 tablet_replica.cc:330] T 70f661b2e85c41638ab202aba9f28bb4 P 9164820a2887463eb2835b4dbb76b505: stopping tablet replica
I20260929 01:03:49.994324 12142 raft_consensus.cc:2229] T 70f661b2e85c41638ab202aba9f28bb4 P 9164820a2887463eb2835b4dbb76b505 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:03:49.994633 12142 raft_consensus.cc:2258] T 70f661b2e85c41638ab202aba9f28bb4 P 9164820a2887463eb2835b4dbb76b505 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:03:49.994719 12143 ts_tablet_manager.cc:1905] T 70f661b2e85c41638ab202aba9f28bb4 P ce2fcf9c2da241a2a3bf16ce8d2e7d8b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 01:03:49.997561 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39477 (error 108)
I20260929 01:03:49.997982 12143 ts_tablet_manager.cc:1918] T 70f661b2e85c41638ab202aba9f28bb4 P ce2fcf9c2da241a2a3bf16ce8d2e7d8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:03:49.998096 12143 log.cc:1196] T 70f661b2e85c41638ab202aba9f28bb4 P ce2fcf9c2da241a2a3bf16ce8d2e7d8b: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-1/wal/wals/70f661b2e85c41638ab202aba9f28bb4
I20260929 01:03:49.998426 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 11835
I20260929 01:03:49.998452 12143 ts_tablet_manager.cc:1939] T 70f661b2e85c41638ab202aba9f28bb4 P ce2fcf9c2da241a2a3bf16ce8d2e7d8b: Deleting consensus metadata
I20260929 01:03:49.999752 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:49.999412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39477 (local address 127.0.0.1:34570)
0929 01:03:49.999561 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:49.999643 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 01:03:49.999910 11599 catalog_manager.cc:4584] TS ad00523ec924434faa0f9f48419bf01f (127.0.98.1:39477): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 70f661b2e85c41638ab202aba9f28bb4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39477: connect: Connection refused (error 111)
I20260929 01:03:50.000882 12142 ts_tablet_manager.cc:1905] T 70f661b2e85c41638ab202aba9f28bb4 P 9164820a2887463eb2835b4dbb76b505: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:03:50.003634 12142 ts_tablet_manager.cc:1918] T 70f661b2e85c41638ab202aba9f28bb4 P 9164820a2887463eb2835b4dbb76b505: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:03:50.003729 12142 log.cc:1196] T 70f661b2e85c41638ab202aba9f28bb4 P 9164820a2887463eb2835b4dbb76b505: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/70f661b2e85c41638ab202aba9f28bb4
I20260929 01:03:50.004045 12142 ts_tablet_manager.cc:1939] T 70f661b2e85c41638ab202aba9f28bb4 P 9164820a2887463eb2835b4dbb76b505: Deleting consensus metadata
I20260929 01:03:50.005409 11602 catalog_manager.cc:4841] TS 9164820a2887463eb2835b4dbb76b505 (127.0.98.3:45281): tablet 70f661b2e85c41638ab202aba9f28bb4 (table test-table [id=8598dc2d94784464bf3f06a59f90c844]) successfully deleted
I20260929 01:03:50.011165 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 11970
I20260929 01:03:50.012023 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:03:50.011396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45979 (local address 127.0.0.1:36736)
0929 01:03:50.011882 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 01:03:50.011936 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392}
I20260929 01:03:50.016878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 11583
I20260929 01:03:50.022178 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 01:03:50.357087 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 01:03:50.357204 392 mini_ranger.cc:66] Stopping Ranger...
I20260929 01:03:50.703611 392 mini_ranger.cc:68] Stopped Ranger
2026-09-29 01:03:50.703 UTC [11169] LOG: received smart shutdown request
2026-09-29 01:03:50.707 UTC [11169] LOG: background worker "logical replication launcher" (PID 11178) exited with exit code 1
2026-09-29 01:03:50.707 UTC [11173] LOG: shutting down
2026-09-29 01:03:50.885 UTC [11169] LOG: database system is shut down
2026-09-29T01:03:50Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45528 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:03:51 dist-test-slave-7zdl krb5kdc[12154](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:03:51 dist-test-slave-7zdl krb5kdc[12154](info): set up 2 sockets
Sep 29 01:03:51 dist-test-slave-7zdl krb5kdc[12154](info): commencing operation
krb5kdc: starting...
W20260929 01:03:52.052213 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:03:52 dist-test-slave-7zdl krb5kdc[12154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:03:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:03:52Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260929 01:03:52.217621 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskS2TZOM/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/postgres -l logfile start
2026-09-29 01:03:54.938 UTC [12184] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 01:03:54.938 UTC [12184] LOG: listening on IPv4 address "127.0.98.20", port 51023
2026-09-29 01:03:54.942 UTC [12184] LOG: listening on Unix socket "/tmp/.s.PGSQL.51023"
2026-09-29 01:03:54.964 UTC [12187] LOG: database system was shut down at 2026-09-29 01:03:53 UTC
2026-09-29 01:03:54.968 UTC [12184] LOG: database system is ready to accept connections
I20260929 01:03:55.934962 392 mini_postgres.cc:96] Postgres bound to 51023
2026-09-29 01:03:55.941 UTC [12195] FATAL: database "slave" does not exist
127.0.98.20:51023 - accepting connections
I20260929 01:03:55.941610 392 mini_ranger.cc:161] Starting Ranger...
I20260929 01:03:55.955776 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 01:03:56.424296 392 mini_ranger.cc:87] Created ranger Postgres database
I20260929 01:03:56.424364 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ranger-admin
2026-09-29 01:03:56,701 [I] DB FLAVOR :POSTGRES
2026-09-29 01:03:56,701 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 01:03:56,701 [I] Checking connection..
2026-09-29 01:03:56,701 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 01:03:56,959 [I] Checking connection passed.
2026-09-29 01:03:56,959 [I] --------- Verifying version history table ---------
2026-09-29 01:03:56,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 01:03:57,218 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 01:03:57,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 01:03:57,475 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 01:03:57,476 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 01:03:57,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51023/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 01:03:57.733 UTC [12259] WARNING: there is no transaction in progress
2026-09-29 01:03:57,739 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 01:03:57,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 01:03:57,975 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 01:03:57,976 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 01:03:57,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 01:03:58,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 01:03:58,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7zdl.c.gcp-upstream.internal','N') ;"
2026-09-29 01:03:58,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 01:03:58,961 [I] Table x_portal_user does not exist in database ranger
2026-09-29 01:03:58,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 01:03:59,213 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 01:03:59,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 01:03:59,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 01:03:59,721 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 01:03:59,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51023/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 01:04:00.676 UTC [12367] WARNING: there is no transaction in progress
2026-09-29 01:04:00.692 UTC [12367] WARNING: there is no transaction in progress
2026-09-29 01:04:00.708 UTC [12367] WARNING: there is no transaction in progress
2026-09-29 01:04:00.726 UTC [12367] WARNING: there is no transaction in progress
2026-09-29 01:04:00.746 UTC [12367] WARNING: there is no transaction in progress
2026-09-29 01:04:00.762 UTC [12367] WARNING: there is no transaction in progress
2026-09-29 01:04:00.817 UTC [12367] WARNING: there is no transaction in progress
2026-09-29 01:04:00.823 UTC [12367] WARNING: there is no transaction in progress
2026-09-29 01:04:00.837 UTC [12367] WARNING: there is no transaction in progress
2026-09-29 01:04:00.847 UTC [12367] WARNING: there is no transaction in progress
2026-09-29 01:04:00.857 UTC [12367] WARNING: there is no transaction in progress
2026-09-29 01:04:00.870 UTC [12367] WARNING: there is no transaction in progress
2026-09-29 01:04:00.879 UTC [12367] WARNING: there is no transaction in progress
2026-09-29 01:04:00.885 UTC [12367] WARNING: there is no transaction in progress
2026-09-29 01:04:00.891 UTC [12367] WARNING: there is no transaction in progress
2026-09-29 01:04:00.897 UTC [12367] WARNING: there is no transaction in progress
2026-09-29 01:04:01,516 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 01:04:01,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 01:04:01,765 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 01:04:01,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 01:04:01,999 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 01:04:02,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 01:04:02,248 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 01:04:02,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 01:04:02,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 01:04:02,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7zdl.c.gcp-upstream.internal';"
2026-09-29 01:04:02,977 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 01:04:02,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 01:04:03,226 [I] DB_PATCHES have already been applied
I20260929 01:04:03.232328 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskS2TZOM/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 01:04:03.232417 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260929 01:04:03.235900 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:37067
Sep 29, 2026 1:04:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 1:04:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 1:04:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 1:04:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 1:04:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 1:04:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 1:04:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 01:04:04.353883 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.412s user 0.000s sys 0.009s
Sep 29, 2026 1:04:24 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 1:04:24 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 1:04:24 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 1:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 1:04:24 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 1:04:25 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 01:04:25.929199 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 01:04:26 dist-test-slave-7zdl krb5kdc[12154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643866, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 01:04:26.033759 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 01:04:26.043318 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 01:04:26.925683 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 01:04:26.925773 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ranger-kms
I20260929 01:04:26.925813 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:51023
2026-09-29 01:04:27,104 [I] DB FLAVOR :POSTGRES
2026-09-29 01:04:27,104 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 01:04:27,105 [I] Checking connection
2026-09-29 01:04:27,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51023/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 01:04:27,334 [I] connection success
2026-09-29 01:04:27,334 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 01:04:27,334 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 01:04:27,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51023/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 01:04:27,557 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 01:04:27,557 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 01:04:27,557 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 01:04:27,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51023/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 01:04:27,817 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 01:04:28.294600 392 mini_ranger_kms.cc:310] Created kms service
I20260929 01:04:28.468703 392 mini_ranger_kms.cc:326] Created kudu user
I20260929 01:04:28.512985 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 01:04:28 dist-test-slave-7zdl krb5kdc[12154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643866, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 01:04:28.897234 392 mini_ranger_kms.cc:384] Added ranger policy
I20260929 01:04:28.897387 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskS2TZOM/build/debug/bin/postgresql.jar:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskS2TZOM/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 01:04:28.897411 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260929 01:04:28.901749 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12547
I20260929 01:04:28.901839 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:49979
Sep 29, 2026 1:04:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 1:04:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 1:04:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 1:04:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 1:04:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 1:04:30 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 01:04:31.208712 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.175s user 0.004s sys 0.159s
I20260929 01:04:31.208843 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 01:04:31.208945 392 mini_ranger_kms.cc:401] 127.0.98.20:49979/kms/v1/keys
Sep 29 01:04:31 dist-test-slave-7zdl krb5kdc[12154](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790643871, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 01:04:31 dist-test-slave-7zdl krb5kdc[12154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643871, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 01:04:32 dist-test-slave-7zdl krb5kdc[12154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643871, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 01:04:34 dist-test-slave-7zdl krb5kdc[12154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:04:34.301314 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38183
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38183
--ranger_config_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49979/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:04:34.413162 12598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:34.413445 12598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:34.413533 12598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:34.417593 12598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:04:34.417665 12598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:34.417702 12598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:34.417757 12598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:04:34.417794 12598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:04:34.423370 12598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:49979/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38183
--ranger_config_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:04:34.424386 12598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:34.425388 12598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:34.434949 12604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:34.434976 12606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:34.434945 12603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:34.435360 12598 server_base.cc:1034] running on GCE node
I20260929 01:04:34.435875 12598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:34.436998 12598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:34.438174 12598 hybrid_clock.cc:648] HybridClock initialized: now 1790643874438161 us; error 48 us; skew 500 ppm
Sep 29 01:04:34 dist-test-slave-7zdl krb5kdc[12154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:34.444479 12598 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:04:34.445366 12598 webserver.cc:460] Webserver started at http://127.0.98.62:37813/ using document root <none> and password file <none>
I20260929 01:04:34.445912 12598 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:34.445991 12598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:34.446201 12598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 01:04:34 dist-test-slave-7zdl krb5kdc[12154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 01:04:34.491715 12598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "be707f8a7d50402ea7b63a44bffe01b7"
format_stamp: "Formatted at 2026-09-29 01:04:34 on dist-test-slave-7zdl"
server_key: "db989b437790480564fa289b0b4795ee"
server_key_iv: "0b8a6490852dc52a7fa56fc627f520f8"
server_key_version: "kuduclusterkey@0"
I20260929 01:04:34.492429 12598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "be707f8a7d50402ea7b63a44bffe01b7"
format_stamp: "Formatted at 2026-09-29 01:04:34 on dist-test-slave-7zdl"
server_key: "db989b437790480564fa289b0b4795ee"
server_key_iv: "0b8a6490852dc52a7fa56fc627f520f8"
server_key_version: "kuduclusterkey@0"
I20260929 01:04:34.497200 12598 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:04:34.513211 12613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:34.514932 12598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:04:34.515062 12598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "be707f8a7d50402ea7b63a44bffe01b7"
format_stamp: "Formatted at 2026-09-29 01:04:34 on dist-test-slave-7zdl"
server_key: "db989b437790480564fa289b0b4795ee"
server_key_iv: "0b8a6490852dc52a7fa56fc627f520f8"
server_key_version: "kuduclusterkey@0"
I20260929 01:04:34.515194 12598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:34.540442 12598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:34.543789 12598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:34.543987 12598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:34.552506 12665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38183 every 8 connection(s)
I20260929 01:04:34.552518 12598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38183
I20260929 01:04:34.553740 12598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:04:34.556555 12666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:34.558066 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 12598
I20260929 01:04:34.558199 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:04:34.558490 392 external_mini_cluster.cc:1428] Setting key f1b2b1695dba622f4ed002b1216dbfc4
I20260929 01:04:34.563324 12666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7: Bootstrap starting.
I20260929 01:04:34.565747 12666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:34.566519 12666 log.cc:824] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:34.568835 12666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7: No bootstrap required, opened a new log
Sep 29 01:04:34 dist-test-slave-7zdl krb5kdc[12154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:04:34.573513 12666 raft_consensus.cc:348] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be707f8a7d50402ea7b63a44bffe01b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38183 } }
I20260929 01:04:34.573827 12666 raft_consensus.cc:374] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:34.573916 12666 raft_consensus.cc:729] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be707f8a7d50402ea7b63a44bffe01b7, State: Initialized, Role: FOLLOWER
I20260929 01:04:34.574443 12666 consensus_queue.cc:260] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be707f8a7d50402ea7b63a44bffe01b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38183 } }
I20260929 01:04:34.574599 12666 raft_consensus.cc:388] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:04:34.574687 12666 raft_consensus.cc:482] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:04:34.574785 12666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:34.575470 12669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:34.559958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48990 (local address 127.0.98.62:38183)
0929 01:04:34.560403 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:34.560422 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:34.560464 (+ 42us) server_negotiation.cc:407] Connection header received
0929 01:04:34.561343 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:34.561367 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:34.561800 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:34.562197 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:34.563056 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:34.564596 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:34.565783 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:34.566574 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:34.566787 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:34.569871 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:34.569878 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:34.569894 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:34.569923 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:34.571995 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:34.572505 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:34.572511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:34.572517 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:34.572648 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:34.572956 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:34.572959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:34.572961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:34.573462 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:34.573678 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:34.574122 (+ 444us) server_negotiation.cc:299] Negotiation successful
0929 01:04:34.574457 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":129,"threads_started":1}
I20260929 01:04:34.575935 12666 raft_consensus.cc:504] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be707f8a7d50402ea7b63a44bffe01b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38183 } }
I20260929 01:04:34.576283 12666 leader_election.cc:302] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be707f8a7d50402ea7b63a44bffe01b7; no voters:
I20260929 01:04:34.576637 12666 leader_election.cc:288] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:04:34.576795 12671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:04:34.577014 12671 raft_consensus.cc:686] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [term 1 LEADER]: Becoming Leader. State: Replica: be707f8a7d50402ea7b63a44bffe01b7, State: Running, Role: LEADER
I20260929 01:04:34.577369 12671 consensus_queue.cc:237] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be707f8a7d50402ea7b63a44bffe01b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38183 } }
I20260929 01:04:34.577992 12666 sys_catalog.cc:564] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:04:34.578595 12671 sys_catalog.cc:455] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be707f8a7d50402ea7b63a44bffe01b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be707f8a7d50402ea7b63a44bffe01b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38183 } } }
I20260929 01:04:34.578842 12671 sys_catalog.cc:458] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:34.579113 12671 sys_catalog.cc:455] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be707f8a7d50402ea7b63a44bffe01b7. Latest consensus state: current_term: 1 leader_uuid: "be707f8a7d50402ea7b63a44bffe01b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be707f8a7d50402ea7b63a44bffe01b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38183 } } }
I20260929 01:04:34.579277 12671 sys_catalog.cc:458] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:34.579589 12678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:04:34.582790 12678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:04:34.583237 12666 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 01:04:34.588825 12678 catalog_manager.cc:1347] Generated new cluster ID: f605ee7a937047ebadc89e2a3d609061
I20260929 01:04:34.588946 12678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:04:34.601181 12678 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:04:34.602933 12678 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:04:34.609826 12678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7: Generated new TSK 0
I20260929 01:04:34.610605 12678 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 01:04:35.310745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49979/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38183
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:04:35.449235 12711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:35.449575 12711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:35.449625 12711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:35.454131 12711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:35.454201 12711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:35.454284 12711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:04:35.461987 12711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:49979/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:04:35.463403 12711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:35.464433 12711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:35.474004 12721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:35.474085 12718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:35.473951 12719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:35.474711 12711 server_base.cc:1034] running on GCE node
I20260929 01:04:35.475122 12711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:35.475808 12711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:35.476994 12711 hybrid_clock.cc:648] HybridClock initialized: now 1790643875476961 us; error 43 us; skew 500 ppm
Sep 29 01:04:35 dist-test-slave-7zdl krb5kdc[12154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:35.480167 12711 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:04:35.481129 12711 webserver.cc:460] Webserver started at http://127.0.98.1:41281/ using document root <none> and password file <none>
I20260929 01:04:35.481688 12711 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:35.481776 12711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:35.481990 12711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 01:04:35 dist-test-slave-7zdl krb5kdc[12154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 01:04:35.495492 12711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "8af14e4b74054f1692879fd3c74f7d1f"
format_stamp: "Formatted at 2026-09-29 01:04:35 on dist-test-slave-7zdl"
server_key: "618ea0c8f6a17823b2b168cd7a72842b"
server_key_iv: "3bfaee3ab10468ce4af7debaa0a86afc"
server_key_version: "kuduclusterkey@0"
I20260929 01:04:35.496287 12711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8af14e4b74054f1692879fd3c74f7d1f"
format_stamp: "Formatted at 2026-09-29 01:04:35 on dist-test-slave-7zdl"
server_key: "618ea0c8f6a17823b2b168cd7a72842b"
server_key_iv: "3bfaee3ab10468ce4af7debaa0a86afc"
server_key_version: "kuduclusterkey@0"
I20260929 01:04:35.504675 12711 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260929 01:04:35.521415 12728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:35.523182 12711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:04:35.523304 12711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "8af14e4b74054f1692879fd3c74f7d1f"
format_stamp: "Formatted at 2026-09-29 01:04:35 on dist-test-slave-7zdl"
server_key: "618ea0c8f6a17823b2b168cd7a72842b"
server_key_iv: "3bfaee3ab10468ce4af7debaa0a86afc"
server_key_version: "kuduclusterkey@0"
I20260929 01:04:35.523461 12711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:35.548368 12711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:35.552207 12711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:35.552448 12711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:35.553197 12711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:35.554278 12711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:35.554382 12711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:35.554505 12711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:35.554600 12711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:35.565014 12711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38763
I20260929 01:04:35.565899 12711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:04:35.571611 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 12711
I20260929 01:04:35.571762 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:04:35.572052 392 external_mini_cluster.cc:1428] Setting key 4ba48ae2dc8b5209989b42e75058ae01
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 01:04:35.583587 12841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38763 every 8 connection(s)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 29 01:04:35 dist-test-slave-7zdl krb5kdc[12154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:04:35.597594 12845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:35.568331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38183 (local address 127.0.98.1:52337)
0929 01:04:35.568797 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:35.568849 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:35.569675 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:35.584555 (+ 14880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:35.584567 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:35.584940 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:35.585669 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:35.585689 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:35.587024 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:35.587028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:35.588464 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:35.588476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:35.589174 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:35.589179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:35.589395 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:35.590183 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:35.590205 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:35.591941 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:35.594294 (+ 2353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:35.594300 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:35.594316 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:35.594640 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:35.595013 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:35.595016 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:35.595018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:35.595191 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:35.595646 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:35.595653 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:35.595990 (+ 337us) client_negotiation.cc:769] Sending connection context
0929 01:04:35.596186 (+ 196us) client_negotiation.cc:240] Negotiation successful
0929 01:04:35.596517 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":135,"threads_started":1}
I20260929 01:04:35.599012 12844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:35.567875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52337 (local address 127.0.98.62:38183)
0929 01:04:35.583938 (+ 16063us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:35.583948 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:35.584011 (+ 63us) server_negotiation.cc:407] Connection header received
0929 01:04:35.584069 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:35.584072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:35.584130 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:35.585845 (+ 1715us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:35.585964 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:35.586866 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:35.588621 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:35.588996 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:35.592111 (+ 3115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:35.592241 (+ 130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:35.592244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:35.592247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:35.592268 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:35.594162 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:35.594789 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:35.594816 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:35.594818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:35.594905 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:35.595314 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:35.595317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:35.595319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:35.595524 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:35.598595 (+ 3071us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:35.598675 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 01:04:35.598836 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15957,"thread_start_us":85,"threads_started":1}
I20260929 01:04:35.600497 12842 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38183
I20260929 01:04:35.600749 12842 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:35.601312 12842 heartbeater.cc:511] Master 127.0.98.62:38183 requested a full tablet report, sending...
I20260929 01:04:35.603188 12630 ts_manager.cc:194] Registered new tserver with Master: 8af14e4b74054f1692879fd3c74f7d1f (127.0.98.1:38763)
I20260929 01:04:35.604951 12630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 01:04:35.610482 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49979/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38183
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:04:35 dist-test-slave-7zdl krb5kdc[12154](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790643875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 01:04:35.768301 12850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:35.768762 12850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:35.768893 12850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:35.773679 12850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:35.773758 12850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:35.773864 12850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:04:35.780249 12850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:49979/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:04:35.781670 12850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:35.783118 12850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:35.795936 12859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:35.796902 12856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:35.797150 12857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:35.797911 12850 server_base.cc:1034] running on GCE node
I20260929 01:04:35.798449 12850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:35.799329 12850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:35.800531 12850 hybrid_clock.cc:648] HybridClock initialized: now 1790643875800462 us; error 88 us; skew 500 ppm
Sep 29 01:04:35 dist-test-slave-7zdl krb5kdc[12154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:35.804302 12850 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:04:35.805371 12850 webserver.cc:460] Webserver started at http://127.0.98.2:36571/ using document root <none> and password file <none>
I20260929 01:04:35.806123 12850 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:35.806250 12850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:35.806532 12850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 01:04:35 dist-test-slave-7zdl krb5kdc[12154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 01:04:35.825485 12850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "6a5050c48dd648d188b9f762bb771a18"
format_stamp: "Formatted at 2026-09-29 01:04:35 on dist-test-slave-7zdl"
server_key: "af0dbb2dba7b6b370cdf6c6fc23d44e5"
server_key_iv: "aa79bf5cbb0975366e57df1d015fbc31"
server_key_version: "kuduclusterkey@0"
I20260929 01:04:35.826181 12850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6a5050c48dd648d188b9f762bb771a18"
format_stamp: "Formatted at 2026-09-29 01:04:35 on dist-test-slave-7zdl"
server_key: "af0dbb2dba7b6b370cdf6c6fc23d44e5"
server_key_iv: "aa79bf5cbb0975366e57df1d015fbc31"
server_key_version: "kuduclusterkey@0"
I20260929 01:04:35.830461 12850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:04:35.851665 12866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:35.853583 12850 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.002s
I20260929 01:04:35.853734 12850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "6a5050c48dd648d188b9f762bb771a18"
format_stamp: "Formatted at 2026-09-29 01:04:35 on dist-test-slave-7zdl"
server_key: "af0dbb2dba7b6b370cdf6c6fc23d44e5"
server_key_iv: "aa79bf5cbb0975366e57df1d015fbc31"
server_key_version: "kuduclusterkey@0"
I20260929 01:04:35.853889 12850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:35.884554 12850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:35.887935 12850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:35.888209 12850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:35.888983 12850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:35.889935 12850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:35.890007 12850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:35.890103 12850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:35.890163 12850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:35.901521 12850 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42031
I20260929 01:04:35.901592 12979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42031 every 8 connection(s)
I20260929 01:04:35.902732 12850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:04:35.906661 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 12850
I20260929 01:04:35.906829 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:04:35.907209 392 external_mini_cluster.cc:1428] Setting key 852791079051411d26f54645e8176ecf
Sep 29 01:04:35 dist-test-slave-7zdl krb5kdc[12154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:04:35.918255 12844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:35.904767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33049 (local address 127.0.98.62:38183)
0929 01:04:35.904933 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:35.904938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:35.905820 (+ 882us) server_negotiation.cc:407] Connection header received
0929 01:04:35.907094 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:35.907098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:35.907171 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:35.907279 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:35.908674 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:35.909258 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:35.910887 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:35.911209 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:35.911285 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:35.914128 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:35.914132 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:35.914135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:35.914166 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:35.916011 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:35.916581 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:35.916588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:35.916594 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:35.916716 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:35.917092 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:35.917095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:35.917097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:35.917287 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:35.917399 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:35.917938 (+ 539us) server_negotiation.cc:299] Negotiation successful
0929 01:04:35.918070 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 01:04:35.919229 12982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:35.905124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38183 (local address 127.0.98.2:33049)
0929 01:04:35.905631 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:35.905688 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:35.906776 (+ 1088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:35.907480 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:35.907488 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:35.907826 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:35.908514 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:35.908534 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:35.909385 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:35.909389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:35.910716 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:35.910727 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:35.911478 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:35.911481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:35.911680 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:35.912301 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:35.912322 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:35.913935 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:35.916137 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:35.916142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:35.916155 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:35.916453 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:35.916820 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:35.916825 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:35.916828 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:35.916976 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:35.917408 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:35.917419 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:35.917690 (+ 271us) client_negotiation.cc:769] Sending connection context
0929 01:04:35.917896 (+ 206us) client_negotiation.cc:240] Negotiation successful
0929 01:04:35.918232 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 01:04:35.920528 12980 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38183
I20260929 01:04:35.920833 12980 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:35.921458 12980 heartbeater.cc:511] Master 127.0.98.62:38183 requested a full tablet report, sending...
I20260929 01:04:35.922596 12630 ts_manager.cc:194] Registered new tserver with Master: 6a5050c48dd648d188b9f762bb771a18 (127.0.98.2:42031)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 01:04:35.923266 12630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 01:04:35.941970 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49979/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38183
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:04:36 dist-test-slave-7zdl krb5kdc[12154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260929 01:04:36.101060 12985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:36.101410 12985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:36.101470 12985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:36.107304 12985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:36.107441 12985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:36.107568 12985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:04:36.115049 12985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:49979/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:04:36.116572 12985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:36.117899 12985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:36.130471 12991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:36.131075 12993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:36.131834 12990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:36.132103 12985 server_base.cc:1034] running on GCE node
I20260929 01:04:36.132637 12985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:36.133332 12985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:36.134500 12985 hybrid_clock.cc:648] HybridClock initialized: now 1790643876134439 us; error 73 us; skew 500 ppm
Sep 29 01:04:36 dist-test-slave-7zdl krb5kdc[12154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:36.137956 12985 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:04:36.138975 12985 webserver.cc:460] Webserver started at http://127.0.98.3:44299/ using document root <none> and password file <none>
I20260929 01:04:36.139627 12985 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:36.139731 12985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:36.139964 12985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 01:04:36 dist-test-slave-7zdl krb5kdc[12154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 01:04:36.168419 12985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "d2f2fd82aba54a73958913d0bb98c81f"
format_stamp: "Formatted at 2026-09-29 01:04:36 on dist-test-slave-7zdl"
server_key: "d3d47df7be007511b762488618a821f9"
server_key_iv: "7753c86ab8ba76d4e81e6b5311810e61"
server_key_version: "kuduclusterkey@0"
I20260929 01:04:36.169087 12985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d2f2fd82aba54a73958913d0bb98c81f"
format_stamp: "Formatted at 2026-09-29 01:04:36 on dist-test-slave-7zdl"
server_key: "d3d47df7be007511b762488618a821f9"
server_key_iv: "7753c86ab8ba76d4e81e6b5311810e61"
server_key_version: "kuduclusterkey@0"
I20260929 01:04:36.173282 12985 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:04:36.192338 13000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:36.194101 12985 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260929 01:04:36.194239 12985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "d2f2fd82aba54a73958913d0bb98c81f"
format_stamp: "Formatted at 2026-09-29 01:04:36 on dist-test-slave-7zdl"
server_key: "d3d47df7be007511b762488618a821f9"
server_key_iv: "7753c86ab8ba76d4e81e6b5311810e61"
server_key_version: "kuduclusterkey@0"
I20260929 01:04:36.194370 12985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:36.216564 12985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:36.219630 12985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:36.219830 12985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:36.220564 12985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:36.221485 12985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:36.221560 12985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:36.221668 12985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:36.221704 12985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:36.232095 12985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45295
I20260929 01:04:36.233623 12985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:04:36.241653 13113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45295 every 8 connection(s)
I20260929 01:04:36.243891 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 12985
I20260929 01:04:36.244017 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:04:36.244338 392 external_mini_cluster.cc:1428] Setting key f9fe57dd942a5f3b9d4862ac32820bd3
Sep 29 01:04:36 dist-test-slave-7zdl krb5kdc[12154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:04:36.259047 12844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:36.245088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34117 (local address 127.0.98.62:38183)
0929 01:04:36.245219 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:36.245223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:36.245867 (+ 644us) server_negotiation.cc:407] Connection header received
0929 01:04:36.247082 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:36.247086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:36.247159 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:36.247221 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:36.248847 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.249682 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.251427 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.251773 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.251853 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:36.254936 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:36.254940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:36.254943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:36.254970 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:36.256710 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:36.257287 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:36.257290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:36.257292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:36.257373 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:36.257753 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:36.257756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:36.257758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:36.257968 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:36.258049 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:36.258717 (+ 668us) server_negotiation.cc:299] Negotiation successful
0929 01:04:36.258877 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 01:04:36.260845 13116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:36.244515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38183 (local address 127.0.98.3:34117)
0929 01:04:36.245710 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:36.245769 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:36.246847 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:36.247640 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:36.247649 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:36.248010 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:36.248681 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:36.248703 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.249838 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.249841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:36.251290 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:36.251300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.252057 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.252061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:36.252272 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:36.252940 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:36.252962 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:36.254729 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:36.256835 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:36.256841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:36.256856 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:36.257177 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:36.257476 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:36.257479 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:36.257481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:36.257640 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:36.258128 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:36.258134 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:36.258436 (+ 302us) client_negotiation.cc:769] Sending connection context
0929 01:04:36.259479 (+ 1043us) client_negotiation.cc:240] Negotiation successful
0929 01:04:36.259803 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"thread_start_us":155,"threads_started":1}
I20260929 01:04:36.262293 13114 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38183
I20260929 01:04:36.262514 13114 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:36.263096 13114 heartbeater.cc:511] Master 127.0.98.62:38183 requested a full tablet report, sending...
I20260929 01:04:36.264299 12630 ts_manager.cc:194] Registered new tserver with Master: d2f2fd82aba54a73958913d0bb98c81f (127.0.98.3:45295)
I20260929 01:04:36.265043 12630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34117
I20260929 01:04:36.269711 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:04:36.270468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 12711
I20260929 01:04:36.279402 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 12850
I20260929 01:04:36.287017 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 12985
I20260929 01:04:36.294533 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 12598
I20260929 01:04:36.300293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38183
--webserver_interface=127.0.98.62
--webserver_port=37813
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38183
--ranger_config_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49979/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:49979/kms with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:04:36.410986 13117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:36.411276 13117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:36.411343 13117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:36.415421 13117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:04:36.415494 13117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:36.415532 13117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:36.415588 13117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:04:36.415633 13117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:04:36.421329 13117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:49979/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38183
--ranger_config_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:04:36.422402 13117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:36.423478 13117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:36.431818 13122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:36.431821 13123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:36.432107 13125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:36.432144 13117 server_base.cc:1034] running on GCE node
I20260929 01:04:36.432682 13117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:36.433772 13117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:36.434995 13117 hybrid_clock.cc:648] HybridClock initialized: now 1790643876434977 us; error 35 us; skew 500 ppm
Sep 29 01:04:36 dist-test-slave-7zdl krb5kdc[12154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:36.437970 13117 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:04:36.439002 13117 webserver.cc:460] Webserver started at http://127.0.98.62:37813/ using document root <none> and password file <none>
I20260929 01:04:36.439554 13117 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:36.439704 13117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 01:04:36 dist-test-slave-7zdl krb5kdc[12154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 01:04:36.492635 13117 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:04:36.495019 13133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:36.496428 13117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:04:36.496556 13117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "be707f8a7d50402ea7b63a44bffe01b7"
format_stamp: "Formatted at 2026-09-29 01:04:34 on dist-test-slave-7zdl"
server_key: "db989b437790480564fa289b0b4795ee"
server_key_iv: "0b8a6490852dc52a7fa56fc627f520f8"
server_key_version: "kuduclusterkey@0"
I20260929 01:04:36.497110 13117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:36.532589 13117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:36.535786 13117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:36.536005 13117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:36.544024 13117 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38183
I20260929 01:04:36.544028 13185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38183 every 8 connection(s)
I20260929 01:04:36.545192 13117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:04:36.547341 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 13117
I20260929 01:04:36.547776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49979/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38763
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38183
--webserver_port=41281
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:49979/kms with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:04:36.550396 13186 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:04:36.552521 13186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7: Bootstrap starting.
I20260929 01:04:36.573603 13186 log.cc:824] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:36.581570 13186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:04:36.582012 13186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7: Bootstrap complete.
I20260929 01:04:36.587669 13186 raft_consensus.cc:348] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be707f8a7d50402ea7b63a44bffe01b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38183 } }
I20260929 01:04:36.588166 13186 raft_consensus.cc:729] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: be707f8a7d50402ea7b63a44bffe01b7, State: Initialized, Role: FOLLOWER
I20260929 01:04:36.588707 13186 consensus_queue.cc:260] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be707f8a7d50402ea7b63a44bffe01b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38183 } }
I20260929 01:04:36.588850 13186 raft_consensus.cc:388] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:04:36.588937 13186 raft_consensus.cc:482] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:04:36.589030 13186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:04:36.591406 13186 raft_consensus.cc:504] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be707f8a7d50402ea7b63a44bffe01b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38183 } }
I20260929 01:04:36.591746 13186 leader_election.cc:302] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be707f8a7d50402ea7b63a44bffe01b7; no voters:
I20260929 01:04:36.592101 13186 leader_election.cc:288] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 01:04:36.592211 13190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:04:36.592511 13190 raft_consensus.cc:686] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [term 2 LEADER]: Becoming Leader. State: Replica: be707f8a7d50402ea7b63a44bffe01b7, State: Running, Role: LEADER
I20260929 01:04:36.592850 13190 consensus_queue.cc:237] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be707f8a7d50402ea7b63a44bffe01b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38183 } }
I20260929 01:04:36.593382 13186 sys_catalog.cc:564] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:04:36.593865 13192 sys_catalog.cc:455] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "be707f8a7d50402ea7b63a44bffe01b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be707f8a7d50402ea7b63a44bffe01b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38183 } } }
I20260929 01:04:36.594003 13192 sys_catalog.cc:458] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:36.593883 13190 sys_catalog.cc:455] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be707f8a7d50402ea7b63a44bffe01b7. Latest consensus state: current_term: 2 leader_uuid: "be707f8a7d50402ea7b63a44bffe01b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be707f8a7d50402ea7b63a44bffe01b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38183 } } }
I20260929 01:04:36.594213 13190 sys_catalog.cc:458] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7 [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:36.594393 13195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:04:36.597460 13195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:04:36.597935 13186 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 01:04:36.598088 13195 catalog_manager.cc:1259] Loaded cluster ID: f605ee7a937047ebadc89e2a3d609061
I20260929 01:04:36.598132 13195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:04:36.600462 13195 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:04:36.601109 13195 catalog_manager.cc:5879] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7: Loaded TSK: 0
I20260929 01:04:36.602285 13195 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 01:04:36.683003 13188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:36.683413 13188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:36.683562 13188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:36.689875 13188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:36.690030 13188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:36.690196 13188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:04:36.698931 13188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:49979/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38763
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41281
--tserver_master_addrs=127.0.98.62:38183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:04:36.700237 13188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:36.701581 13188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:36.714221 13224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:36.714251 13223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:36.714417 13226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:36.714607 13188 server_base.cc:1034] running on GCE node
I20260929 01:04:36.715286 13188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:36.716104 13188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:36.717311 13188 hybrid_clock.cc:648] HybridClock initialized: now 1790643876717285 us; error 45 us; skew 500 ppm
Sep 29 01:04:36 dist-test-slave-7zdl krb5kdc[12154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:36.723260 13188 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:04:36.724524 13188 webserver.cc:460] Webserver started at http://127.0.98.1:41281/ using document root <none> and password file <none>
I20260929 01:04:36.725337 13188 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:36.725502 13188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 01:04:36 dist-test-slave-7zdl krb5kdc[12154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 01:04:36.765348 13188 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:04:36.768558 13234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:36.770231 13188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:04:36.770417 13188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "8af14e4b74054f1692879fd3c74f7d1f"
format_stamp: "Formatted at 2026-09-29 01:04:35 on dist-test-slave-7zdl"
server_key: "618ea0c8f6a17823b2b168cd7a72842b"
server_key_iv: "3bfaee3ab10468ce4af7debaa0a86afc"
server_key_version: "kuduclusterkey@0"
I20260929 01:04:36.771256 13188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:36.798416 13188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:36.802075 13188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:36.802299 13188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:36.803076 13188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:36.804018 13188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:36.804083 13188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:36.804137 13188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:36.804203 13188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 01:04:36.815032 13188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38763
I20260929 01:04:36.815150 13347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38763 every 8 connection(s)
I20260929 01:04:36.816179 13188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:04:36.818049 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 13188
I20260929 01:04:36.818498 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49979/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42031
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38183
--webserver_port=36571
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:49979/kms with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:04:36 dist-test-slave-7zdl krb5kdc[12154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:04:36.838102 13351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:36.819342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40949 (local address 127.0.98.62:38183)
0929 01:04:36.820449 (+ 1107us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:36.820468 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:36.820843 (+ 375us) server_negotiation.cc:407] Connection header received
0929 01:04:36.822118 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:36.822134 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:36.822574 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:36.822975 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:36.824735 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.826303 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.828007 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.828757 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.828912 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:36.832112 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:36.832118 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:36.832134 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:36.832166 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:36.834320 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:36.834983 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:36.834989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:36.834994 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:36.835132 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:36.835548 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:36.835551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:36.835553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:36.836008 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:36.836138 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:36.836876 (+ 738us) server_negotiation.cc:299] Negotiation successful
0929 01:04:36.837164 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":932,"thread_start_us":106,"threads_started":1}
I20260929 01:04:36.838147 13350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:36.818557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38183 (local address 127.0.98.1:40949)
0929 01:04:36.820676 (+ 2119us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:36.820714 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:36.821590 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:36.823531 (+ 1941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:36.823539 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:36.823893 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:36.824533 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:36.824553 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.826461 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.826465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:36.827860 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:36.827872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:36.829128 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:36.829132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:36.829334 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:36.830027 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:36.830049 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:36.831888 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:36.834455 (+ 2567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:36.834463 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:36.834479 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:36.834846 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:36.835265 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:36.835269 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:36.835271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:36.835439 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:36.836205 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:36.836211 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:36.836506 (+ 295us) client_negotiation.cc:769] Sending connection context
0929 01:04:36.836696 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 01:04:36.837023 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1950,"thread_start_us":1044,"threads_started":1}
I20260929 01:04:36.840826 13348 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38183
I20260929 01:04:36.841099 13348 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:36.841697 13348 heartbeater.cc:511] Master 127.0.98.62:38183 requested a full tablet report, sending...
W20260929 01:04:36.844082 13348 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38183 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 01:04:36.844220 13348 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:36.844379 13348 heartbeater.cc:511] Master 127.0.98.62:38183 requested a full tablet report, sending...
I20260929 01:04:36.845338 13348 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:36.845510 13348 heartbeater.cc:511] Master 127.0.98.62:38183 requested a full tablet report, sending...
W20260929 01:04:36.846323 13348 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 01:04:36.960096 13352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:36.960362 13352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:36.960474 13352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:36.964931 13352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:36.965021 13352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:36.965140 13352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:04:36.971480 13352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:49979/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36571
--tserver_master_addrs=127.0.98.62:38183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:04:36.972525 13352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:36.973623 13352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:36.984899 13360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:36.985096 13357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:36.984905 13358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:36.985527 13352 server_base.cc:1034] running on GCE node
I20260929 01:04:36.986001 13352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:36.986840 13352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:36.988045 13352 hybrid_clock.cc:648] HybridClock initialized: now 1790643876988012 us; error 43 us; skew 500 ppm
Sep 29 01:04:36 dist-test-slave-7zdl krb5kdc[12154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:36.991700 13352 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:04:36.992889 13352 webserver.cc:460] Webserver started at http://127.0.98.2:36571/ using document root <none> and password file <none>
I20260929 01:04:36.993810 13352 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:36.993930 13352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 01:04:36 dist-test-slave-7zdl krb5kdc[12154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 01:04:37.017434 13352 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:04:37.026156 13368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:37.027957 13352 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260929 01:04:37.028119 13352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "6a5050c48dd648d188b9f762bb771a18"
format_stamp: "Formatted at 2026-09-29 01:04:35 on dist-test-slave-7zdl"
server_key: "af0dbb2dba7b6b370cdf6c6fc23d44e5"
server_key_iv: "aa79bf5cbb0975366e57df1d015fbc31"
server_key_version: "kuduclusterkey@0"
I20260929 01:04:37.028882 13352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:37.055672 13352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:37.059382 13352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:37.059615 13352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:37.060405 13352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:37.061553 13352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:37.061619 13352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:37.061676 13352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:37.061702 13352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:37.074254 13352 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42031
I20260929 01:04:37.074292 13481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42031 every 8 connection(s)
I20260929 01:04:37.075613 13352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:04:37.078372 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 13352
I20260929 01:04:37.078873 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49979/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45295
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38183
--webserver_port=44299
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:49979/kms with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:04:37 dist-test-slave-7zdl krb5kdc[12154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:04:37.095221 13351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:37.079681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34395 (local address 127.0.98.62:38183)
0929 01:04:37.079841 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:37.079846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:37.080307 (+ 461us) server_negotiation.cc:407] Connection header received
0929 01:04:37.081634 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:37.081638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:37.081696 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:37.081798 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:37.083338 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:37.084172 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:37.085915 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:37.086292 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:37.086403 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:37.090260 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:37.090267 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:37.090274 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:37.090312 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:37.092435 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:37.093110 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:37.093117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:37.093122 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:37.093215 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:37.093674 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:37.093680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:37.093684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:37.093951 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:37.094050 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:37.094791 (+ 741us) server_negotiation.cc:299] Negotiation successful
0929 01:04:37.094965 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 01:04:37.097198 13484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:37.078599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38183 (local address 127.0.98.2:34395)
0929 01:04:37.080131 (+ 1532us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:37.080189 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:37.081367 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:37.081999 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:37.082008 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:37.082376 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:37.083118 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:37.083148 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:37.084358 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:37.084362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:37.085771 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:37.085783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:37.086580 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:37.086584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:37.086775 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:37.088360 (+ 1585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:37.088386 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:37.090012 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:37.092579 (+ 2567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:37.092587 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:37.092604 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:37.092978 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:37.093355 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:37.093361 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:37.093364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:37.093557 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:37.094125 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:37.094135 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:37.094481 (+ 346us) client_negotiation.cc:769] Sending connection context
0929 01:04:37.095664 (+ 1183us) client_negotiation.cc:240] Negotiation successful
0929 01:04:37.096042 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1295,"thread_start_us":168,"threads_started":1}
I20260929 01:04:37.098722 13482 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38183
I20260929 01:04:37.099046 13482 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:37.099738 13482 heartbeater.cc:511] Master 127.0.98.62:38183 requested a full tablet report, sending...
W20260929 01:04:37.101557 13482 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38183 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 01:04:37.101694 13482 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:37.101892 13482 heartbeater.cc:511] Master 127.0.98.62:38183 requested a full tablet report, sending...
I20260929 01:04:37.102787 13482 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:37.103026 13482 heartbeater.cc:511] Master 127.0.98.62:38183 requested a full tablet report, sending...
W20260929 01:04:37.103873 13482 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 01:04:37.232532 13485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:37.232928 13485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:37.233021 13485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:37.238531 13485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:37.238605 13485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:37.238695 13485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:04:37.245690 13485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:49979/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45295
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44299
--tserver_master_addrs=127.0.98.62:38183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:04:37.246846 13485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:37.248042 13485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:37.258839 13492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:37.259073 13494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:37.258836 13491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:37.260222 13485 server_base.cc:1034] running on GCE node
I20260929 01:04:37.260710 13485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:37.261590 13485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:37.262789 13485 hybrid_clock.cc:648] HybridClock initialized: now 1790643877262755 us; error 43 us; skew 500 ppm
Sep 29 01:04:37 dist-test-slave-7zdl krb5kdc[12154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:37.266399 13485 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:04:37.267442 13485 webserver.cc:460] Webserver started at http://127.0.98.3:44299/ using document root <none> and password file <none>
I20260929 01:04:37.268131 13485 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:37.268258 13485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 01:04:37 dist-test-slave-7zdl krb5kdc[12154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 01:04:37.304483 13485 fs_manager.cc:709] Time spent opening directory manager: real 0.017s user 0.003s sys 0.001s
I20260929 01:04:37.311650 13502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:37.313342 13485 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.001s
I20260929 01:04:37.313463 13485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "d2f2fd82aba54a73958913d0bb98c81f"
format_stamp: "Formatted at 2026-09-29 01:04:36 on dist-test-slave-7zdl"
server_key: "d3d47df7be007511b762488618a821f9"
server_key_iv: "7753c86ab8ba76d4e81e6b5311810e61"
server_key_version: "kuduclusterkey@0"
I20260929 01:04:37.314168 13485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:37.359951 13485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:37.363559 13485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:37.363766 13485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:37.364382 13485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:37.365389 13485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:37.365501 13485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:37.365607 13485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:37.365674 13485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:37.377029 13485 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45295
I20260929 01:04:37.377159 13615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45295 every 8 connection(s)
I20260929 01:04:37.378484 13485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:04:37.382839 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 13485
Sep 29 01:04:37 dist-test-slave-7zdl krb5kdc[12154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:04:37.396585 13351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:37.382402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37301 (local address 127.0.98.62:38183)
0929 01:04:37.382597 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:37.382602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:37.383663 (+ 1061us) server_negotiation.cc:407] Connection header received
0929 01:04:37.384961 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:37.384964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:37.385019 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:37.385097 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:37.386687 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:37.387445 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:37.389160 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:37.389496 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:37.389567 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:37.392394 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:37.392398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:37.392401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:37.392426 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:37.394264 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:37.394901 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:37.394904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:37.394906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:37.394991 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:37.395369 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:37.395372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:37.395374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:37.395575 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:37.395653 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:37.396291 (+ 638us) server_negotiation.cc:299] Negotiation successful
0929 01:04:37.396416 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 01:04:37.398252 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:37.382981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38183 (local address 127.0.98.3:37301)
0929 01:04:37.383490 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:37.383543 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:37.384740 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:37.385413 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:37.385424 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:37.385787 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:37.386512 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:37.386531 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:37.387626 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:37.387629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:37.389021 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:37.389030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:37.389753 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:37.389757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:37.389966 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:37.390640 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:37.390661 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:37.392210 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:37.394392 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:37.394398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:37.394412 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:37.394749 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:37.395101 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:37.395103 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:37.395106 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:37.395263 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:37.395714 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:37.395720 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:37.396021 (+ 301us) client_negotiation.cc:769] Sending connection context
0929 01:04:37.397141 (+ 1120us) client_negotiation.cc:240] Negotiation successful
0929 01:04:37.397452 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":136,"threads_started":1}
I20260929 01:04:37.399498 13616 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38183
I20260929 01:04:37.399739 13616 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:37.400327 13616 heartbeater.cc:511] Master 127.0.98.62:38183 requested a full tablet report, sending...
W20260929 01:04:37.402062 13616 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38183 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 01:04:37.402187 13616 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:37.402370 13616 heartbeater.cc:511] Master 127.0.98.62:38183 requested a full tablet report, sending...
I20260929 01:04:37.403254 13616 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:37.403435 13616 heartbeater.cc:511] Master 127.0.98.62:38183 requested a full tablet report, sending...
W20260929 01:04:37.404210 13616 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 01:04:37.411574 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:37.384176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49000 (local address 127.0.98.62:38183)
0929 01:04:37.385145 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:37.385154 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:37.385427 (+ 273us) server_negotiation.cc:407] Connection header received
0929 01:04:37.385613 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:37.385616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:37.385669 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:37.385732 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:37.386737 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:37.387580 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:37.397085 (+ 9505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:37.397419 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:37.397528 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:37.404410 (+ 6882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:37.404415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:37.404430 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:37.404457 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:37.406303 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:37.410165 (+ 3862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:37.410168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:37.410170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:37.410284 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:37.410618 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:37.410625 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:37.410626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:37.410893 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:37.411110 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:37.411250 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 01:04:37.411389 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":906,"thread_start_us":88,"threads_started":1}
Sep 29 01:04:37 dist-test-slave-7zdl krb5kdc[12154](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790643877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:37.846539 13348 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:37.846897 13348 heartbeater.cc:511] Master 127.0.98.62:38183 requested a full tablet report, sending...
I20260929 01:04:37.847981 13150 ts_manager.cc:194] Registered new tserver with Master: 8af14e4b74054f1692879fd3c74f7d1f (127.0.98.1:38763)
I20260929 01:04:37.849210 13150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40949
I20260929 01:04:38.104132 13482 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:38.104473 13482 heartbeater.cc:511] Master 127.0.98.62:38183 requested a full tablet report, sending...
I20260929 01:04:38.105330 13150 ts_manager.cc:194] Registered new tserver with Master: 6a5050c48dd648d188b9f762bb771a18 (127.0.98.2:42031)
I20260929 01:04:38.106158 13150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34395
Sep 29 01:04:38 dist-test-slave-7zdl krb5kdc[12154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 01:04:38.404471 13616 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:38.404806 13616 heartbeater.cc:511] Master 127.0.98.62:38183 requested a full tablet report, sending...
I20260929 01:04:38.405699 13150 ts_manager.cc:194] Registered new tserver with Master: d2f2fd82aba54a73958913d0bb98c81f (127.0.98.3:45295)
I20260929 01:04:38.406445 13150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37301
I20260929 01:04:38.410208 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:04:38.424225 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:38.411911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49008 (local address 127.0.98.62:38183)
0929 01:04:38.412339 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:38.412343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:38.412672 (+ 329us) server_negotiation.cc:407] Connection header received
0929 01:04:38.412713 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:38.412716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:38.412758 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:38.412827 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:38.413703 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:38.414539 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:38.415703 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:38.416033 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:38.416125 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:38.417831 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:38.417835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:38.417838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:38.417862 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:38.422171 (+ 4309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:38.422707 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:38.422711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:38.422713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:38.422855 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:38.423168 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:38.423172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:38.423174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:38.423428 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:38.423572 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:38.423904 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 01:04:38.424041 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":85,"threads_started":1}
I20260929 01:04:38.428520 13150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
Sep 29 01:04:38 dist-test-slave-7zdl krb5kdc[12154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260929 01:04:38.433337 13150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:04:38.481544 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:38.471589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45295 (local address 127.0.0.1:60248)
0929 01:04:38.475115 (+ 3526us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:38.475127 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:38.475204 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:38.476701 (+ 1497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:38.476705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:38.476728 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:38.476839 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:38.476845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:38.478490 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:38.478493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:38.479775 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:38.479789 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:38.481068 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:38.481072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:38.481188 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:38.481261 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 01:04:38.481331 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 01:04:38.481400 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2799,"mutex_wait_us":39,"thread_start_us":92,"threads_started":1}
I20260929 01:04:38.482071 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:38.471757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60248 (local address 127.0.98.3:45295)
0929 01:04:38.476263 (+ 4506us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:38.476270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:38.476293 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:04:38.476361 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:38.476365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:38.476541 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:38.476694 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:38.476987 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:38.478356 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:38.479937 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:38.480760 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:38.480875 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:38.481007 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:38.481835 (+ 828us) server_negotiation.cc:299] Negotiation successful
0929 01:04:38.481936 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4365,"thread_start_us":94,"threads_started":1}
I20260929 01:04:38.482457 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:38.470526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38763 (local address 127.0.0.1:54170)
0929 01:04:38.473161 (+ 2635us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:38.473172 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:38.473234 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:38.474107 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:38.474113 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:38.474136 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:38.474254 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:38.474260 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:38.477414 (+ 3154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:38.477418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:38.479516 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:38.479525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:38.480813 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:38.480817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:38.480916 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:38.480967 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 01:04:38.481068 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 01:04:38.481186 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2579,"mutex_wait_us":60,"thread_start_us":98,"threads_started":1}
I20260929 01:04:38.483143 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:38.470947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54170 (local address 127.0.98.1:38763)
0929 01:04:38.473468 (+ 2521us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:38.473475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:38.473498 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:04:38.473569 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:38.473574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:38.473796 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:38.473938 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:38.475308 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:38.477288 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:38.479681 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:38.480502 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:38.480602 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:38.480756 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:38.482903 (+ 2147us) server_negotiation.cc:299] Negotiation successful
0929 01:04:38.483003 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2417,"thread_start_us":78,"threads_started":1}
I20260929 01:04:38.483842 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:38.469137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42031 (local address 127.0.0.1:46190)
0929 01:04:38.472787 (+ 3650us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:38.472838 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:38.472960 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:38.479651 (+ 6691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:38.479654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:38.479672 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:38.479760 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:38.479766 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:38.481050 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:38.481056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:38.482176 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:38.482186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:38.483483 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:38.483487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:38.483573 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:38.483594 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 01:04:38.483666 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 01:04:38.483727 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3475,"mutex_wait_us":52,"thread_start_us":95,"threads_started":1}
I20260929 01:04:38.484371 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:38.468820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46190 (local address 127.0.98.2:42031)
0929 01:04:38.479038 (+ 10218us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:38.479045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:38.479066 (+ 21us) server_negotiation.cc:407] Connection header received
0929 01:04:38.479146 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:38.479151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:38.479333 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:38.479477 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:38.479886 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:38.480907 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:38.482306 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:38.483149 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:38.483261 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:38.483422 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:38.484167 (+ 745us) server_negotiation.cc:299] Negotiation successful
0929 01:04:38.484266 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10116,"thread_start_us":126,"threads_started":1}
I20260929 01:04:38.485373 13550 tablet_service.cc:1467] Processing CreateTablet for tablet bd018814d67f4de6b19a165350092680 (DEFAULT_TABLE table=test-table [id=e8738d9148ce4e199bd3bfe02c341bce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:04:38.486316 13550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd018814d67f4de6b19a165350092680. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:38.487150 13398 tablet_service.cc:1467] Processing CreateTablet for tablet bd018814d67f4de6b19a165350092680 (DEFAULT_TABLE table=test-table [id=e8738d9148ce4e199bd3bfe02c341bce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:04:38.488117 13398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd018814d67f4de6b19a165350092680. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:38.492797 13282 tablet_service.cc:1467] Processing CreateTablet for tablet bd018814d67f4de6b19a165350092680 (DEFAULT_TABLE table=test-table [id=e8738d9148ce4e199bd3bfe02c341bce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:04:38.493763 13282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd018814d67f4de6b19a165350092680. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:38.494462 13651 tablet_bootstrap.cc:492] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f: Bootstrap starting.
I20260929 01:04:38.497323 13651 tablet_bootstrap.cc:654] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:38.498243 13651 log.cc:824] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:38.499404 13652 tablet_bootstrap.cc:492] T bd018814d67f4de6b19a165350092680 P 6a5050c48dd648d188b9f762bb771a18: Bootstrap starting.
I20260929 01:04:38.500468 13651 tablet_bootstrap.cc:492] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f: No bootstrap required, opened a new log
I20260929 01:04:38.500649 13651 ts_tablet_manager.cc:1397] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 01:04:38.502635 13652 tablet_bootstrap.cc:654] T bd018814d67f4de6b19a165350092680 P 6a5050c48dd648d188b9f762bb771a18: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:38.503453 13652 log.cc:824] T bd018814d67f4de6b19a165350092680 P 6a5050c48dd648d188b9f762bb771a18: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:38.505488 13652 tablet_bootstrap.cc:492] T bd018814d67f4de6b19a165350092680 P 6a5050c48dd648d188b9f762bb771a18: No bootstrap required, opened a new log
I20260929 01:04:38.505686 13652 ts_tablet_manager.cc:1397] T bd018814d67f4de6b19a165350092680 P 6a5050c48dd648d188b9f762bb771a18: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260929 01:04:38.506064 13651 raft_consensus.cc:348] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5050c48dd648d188b9f762bb771a18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42031 } } peers { permanent_uuid: "8af14e4b74054f1692879fd3c74f7d1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38763 } } peers { permanent_uuid: "d2f2fd82aba54a73958913d0bb98c81f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45295 } }
I20260929 01:04:38.506333 13651 raft_consensus.cc:374] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:38.506387 13651 raft_consensus.cc:729] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f2fd82aba54a73958913d0bb98c81f, State: Initialized, Role: FOLLOWER
I20260929 01:04:38.506932 13651 consensus_queue.cc:260] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5050c48dd648d188b9f762bb771a18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42031 } } peers { permanent_uuid: "8af14e4b74054f1692879fd3c74f7d1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38763 } } peers { permanent_uuid: "d2f2fd82aba54a73958913d0bb98c81f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45295 } }
I20260929 01:04:38.507983 13651 ts_tablet_manager.cc:1428] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260929 01:04:38.508342 13616 heartbeater.cc:503] Master 127.0.98.62:38183 was elected leader, sending a full tablet report...
I20260929 01:04:38.511399 13652 raft_consensus.cc:348] T bd018814d67f4de6b19a165350092680 P 6a5050c48dd648d188b9f762bb771a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5050c48dd648d188b9f762bb771a18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42031 } } peers { permanent_uuid: "8af14e4b74054f1692879fd3c74f7d1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38763 } } peers { permanent_uuid: "d2f2fd82aba54a73958913d0bb98c81f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45295 } }
I20260929 01:04:38.511812 13652 raft_consensus.cc:374] T bd018814d67f4de6b19a165350092680 P 6a5050c48dd648d188b9f762bb771a18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:38.511900 13652 raft_consensus.cc:729] T bd018814d67f4de6b19a165350092680 P 6a5050c48dd648d188b9f762bb771a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a5050c48dd648d188b9f762bb771a18, State: Initialized, Role: FOLLOWER
I20260929 01:04:38.512422 13652 consensus_queue.cc:260] T bd018814d67f4de6b19a165350092680 P 6a5050c48dd648d188b9f762bb771a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5050c48dd648d188b9f762bb771a18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42031 } } peers { permanent_uuid: "8af14e4b74054f1692879fd3c74f7d1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38763 } } peers { permanent_uuid: "d2f2fd82aba54a73958913d0bb98c81f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45295 } }
I20260929 01:04:38.513836 13652 ts_tablet_manager.cc:1428] T bd018814d67f4de6b19a165350092680 P 6a5050c48dd648d188b9f762bb771a18: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260929 01:04:38.514195 13482 heartbeater.cc:503] Master 127.0.98.62:38183 was elected leader, sending a full tablet report...
I20260929 01:04:38.523227 13658 tablet_bootstrap.cc:492] T bd018814d67f4de6b19a165350092680 P 8af14e4b74054f1692879fd3c74f7d1f: Bootstrap starting.
I20260929 01:04:38.528013 13658 tablet_bootstrap.cc:654] T bd018814d67f4de6b19a165350092680 P 8af14e4b74054f1692879fd3c74f7d1f: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:38.530011 13658 log.cc:824] T bd018814d67f4de6b19a165350092680 P 8af14e4b74054f1692879fd3c74f7d1f: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:38.532917 13658 tablet_bootstrap.cc:492] T bd018814d67f4de6b19a165350092680 P 8af14e4b74054f1692879fd3c74f7d1f: No bootstrap required, opened a new log
I20260929 01:04:38.533146 13658 ts_tablet_manager.cc:1397] T bd018814d67f4de6b19a165350092680 P 8af14e4b74054f1692879fd3c74f7d1f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260929 01:04:38.538022 13658 raft_consensus.cc:348] T bd018814d67f4de6b19a165350092680 P 8af14e4b74054f1692879fd3c74f7d1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5050c48dd648d188b9f762bb771a18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42031 } } peers { permanent_uuid: "8af14e4b74054f1692879fd3c74f7d1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38763 } } peers { permanent_uuid: "d2f2fd82aba54a73958913d0bb98c81f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45295 } }
I20260929 01:04:38.538396 13658 raft_consensus.cc:374] T bd018814d67f4de6b19a165350092680 P 8af14e4b74054f1692879fd3c74f7d1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:38.538488 13658 raft_consensus.cc:729] T bd018814d67f4de6b19a165350092680 P 8af14e4b74054f1692879fd3c74f7d1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8af14e4b74054f1692879fd3c74f7d1f, State: Initialized, Role: FOLLOWER
I20260929 01:04:38.539108 13658 consensus_queue.cc:260] T bd018814d67f4de6b19a165350092680 P 8af14e4b74054f1692879fd3c74f7d1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5050c48dd648d188b9f762bb771a18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42031 } } peers { permanent_uuid: "8af14e4b74054f1692879fd3c74f7d1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38763 } } peers { permanent_uuid: "d2f2fd82aba54a73958913d0bb98c81f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45295 } }
I20260929 01:04:38.540287 13658 ts_tablet_manager.cc:1428] T bd018814d67f4de6b19a165350092680 P 8af14e4b74054f1692879fd3c74f7d1f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 01:04:38.540745 13348 heartbeater.cc:503] Master 127.0.98.62:38183 was elected leader, sending a full tablet report...
W20260929 01:04:38.570276 13349 tablet.cc:2369] T bd018814d67f4de6b19a165350092680 P 8af14e4b74054f1692879fd3c74f7d1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:04:38.578361 13483 tablet.cc:2369] T bd018814d67f4de6b19a165350092680 P 6a5050c48dd648d188b9f762bb771a18: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:04:38.633174 13617 tablet.cc:2369] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:04:38.771615 13656 raft_consensus.cc:482] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:04:38.771907 13656 raft_consensus.cc:504] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5050c48dd648d188b9f762bb771a18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42031 } } peers { permanent_uuid: "8af14e4b74054f1692879fd3c74f7d1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38763 } } peers { permanent_uuid: "d2f2fd82aba54a73958913d0bb98c81f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45295 } }
I20260929 01:04:38.773500 13656 leader_election.cc:288] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a5050c48dd648d188b9f762bb771a18 (127.0.98.2:42031), 8af14e4b74054f1692879fd3c74f7d1f (127.0.98.1:38763)
I20260929 01:04:38.778723 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:38.774047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43845 (local address 127.0.98.2:42031)
0929 01:04:38.774220 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:38.774225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:38.774358 (+ 133us) server_negotiation.cc:407] Connection header received
0929 01:04:38.774509 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:38.774516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:38.774584 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:38.774662 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:38.775039 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:38.775722 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:38.777429 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:38.778088 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:38.778155 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:38.778210 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:38.778507 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 01:04:38.778583 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 01:04:38.778764 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:38.773700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42031 (local address 127.0.98.3:43845)
0929 01:04:38.774241 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:38.774258 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:38.774383 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:38.774694 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:38.774698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:38.774719 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:38.774906 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:38.774914 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:38.775837 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:38.775844 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:38.777283 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:38.777295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:38.778294 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:38.778297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:38.778373 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:38.778392 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 01:04:38.778487 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 01:04:38.778565 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":7,"thread_start_us":136,"threads_started":1}
I20260929 01:04:38.779245 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:38.774080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36537 (local address 127.0.98.1:38763)
0929 01:04:38.774230 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:38.774238 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:38.774600 (+ 362us) server_negotiation.cc:407] Connection header received
0929 01:04:38.774781 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:38.774785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:38.774871 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:38.774926 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:38.775370 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:38.776144 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:38.777738 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:38.778406 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:38.778512 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:38.778576 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:38.779037 (+ 461us) server_negotiation.cc:299] Negotiation successful
0929 01:04:38.779124 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 01:04:38.779461 13418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd018814d67f4de6b19a165350092680" candidate_uuid: "d2f2fd82aba54a73958913d0bb98c81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a5050c48dd648d188b9f762bb771a18" is_pre_election: true
I20260929 01:04:38.779680 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:38.773867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38763 (local address 127.0.98.3:36537)
0929 01:04:38.774475 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:38.774489 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:38.774599 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:38.775084 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:38.775087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:38.775104 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:38.775229 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:38.775236 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:38.776315 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:38.776319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:38.777588 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:38.777597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:38.778636 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:38.778640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:38.778745 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:38.778770 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 01:04:38.779458 (+ 688us) client_negotiation.cc:240] Negotiation successful
0929 01:04:38.779531 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":36,"thread_start_us":130,"threads_started":1}
I20260929 01:04:38.779863 13418 raft_consensus.cc:2454] T bd018814d67f4de6b19a165350092680 P 6a5050c48dd648d188b9f762bb771a18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2f2fd82aba54a73958913d0bb98c81f in term 0.
I20260929 01:04:38.780356 13302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd018814d67f4de6b19a165350092680" candidate_uuid: "d2f2fd82aba54a73958913d0bb98c81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8af14e4b74054f1692879fd3c74f7d1f" is_pre_election: true
I20260929 01:04:38.780373 13503 leader_election.cc:302] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a5050c48dd648d188b9f762bb771a18, d2f2fd82aba54a73958913d0bb98c81f; no voters:
I20260929 01:04:38.780653 13302 raft_consensus.cc:2454] T bd018814d67f4de6b19a165350092680 P 8af14e4b74054f1692879fd3c74f7d1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2f2fd82aba54a73958913d0bb98c81f in term 0.
I20260929 01:04:38.780704 13656 raft_consensus.cc:2781] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:04:38.780791 13656 raft_consensus.cc:482] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:04:38.780846 13656 raft_consensus.cc:3037] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:38.781947 13656 raft_consensus.cc:504] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5050c48dd648d188b9f762bb771a18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42031 } } peers { permanent_uuid: "8af14e4b74054f1692879fd3c74f7d1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38763 } } peers { permanent_uuid: "d2f2fd82aba54a73958913d0bb98c81f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45295 } }
I20260929 01:04:38.782349 13656 leader_election.cc:288] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f [CANDIDATE]: Term 1 election: Requested vote from peers 6a5050c48dd648d188b9f762bb771a18 (127.0.98.2:42031), 8af14e4b74054f1692879fd3c74f7d1f (127.0.98.1:38763)
I20260929 01:04:38.782704 13418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd018814d67f4de6b19a165350092680" candidate_uuid: "d2f2fd82aba54a73958913d0bb98c81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a5050c48dd648d188b9f762bb771a18"
I20260929 01:04:38.782873 13418 raft_consensus.cc:3037] T bd018814d67f4de6b19a165350092680 P 6a5050c48dd648d188b9f762bb771a18 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:38.782881 13302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd018814d67f4de6b19a165350092680" candidate_uuid: "d2f2fd82aba54a73958913d0bb98c81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8af14e4b74054f1692879fd3c74f7d1f"
I20260929 01:04:38.783047 13302 raft_consensus.cc:3037] T bd018814d67f4de6b19a165350092680 P 8af14e4b74054f1692879fd3c74f7d1f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:38.784166 13302 raft_consensus.cc:2454] T bd018814d67f4de6b19a165350092680 P 8af14e4b74054f1692879fd3c74f7d1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2f2fd82aba54a73958913d0bb98c81f in term 1.
I20260929 01:04:38.784243 13418 raft_consensus.cc:2454] T bd018814d67f4de6b19a165350092680 P 6a5050c48dd648d188b9f762bb771a18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2f2fd82aba54a73958913d0bb98c81f in term 1.
I20260929 01:04:38.784507 13505 leader_election.cc:302] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8af14e4b74054f1692879fd3c74f7d1f, d2f2fd82aba54a73958913d0bb98c81f; no voters:
I20260929 01:04:38.784694 13656 raft_consensus.cc:2781] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:04:38.784964 13656 raft_consensus.cc:686] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f [term 1 LEADER]: Becoming Leader. State: Replica: d2f2fd82aba54a73958913d0bb98c81f, State: Running, Role: LEADER
I20260929 01:04:38.785329 13656 consensus_queue.cc:237] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5050c48dd648d188b9f762bb771a18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42031 } } peers { permanent_uuid: "8af14e4b74054f1692879fd3c74f7d1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38763 } } peers { permanent_uuid: "d2f2fd82aba54a73958913d0bb98c81f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45295 } }
I20260929 01:04:38.789074 13150 catalog_manager.cc:5495] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f reported cstate change: term changed from 0 to 1, leader changed from <none> to d2f2fd82aba54a73958913d0bb98c81f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d2f2fd82aba54a73958913d0bb98c81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a5050c48dd648d188b9f762bb771a18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8af14e4b74054f1692879fd3c74f7d1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2f2fd82aba54a73958913d0bb98c81f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45295 } health_report { overall_health: HEALTHY } } }
I20260929 01:04:38.852887 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:38.849513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60254 (local address 127.0.98.3:45295)
0929 01:04:38.849643 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:38.849646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:38.849659 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:04:38.849778 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:38.849781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:38.849844 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:38.849937 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:04:38.850207 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:38.850698 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:38.851596 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:38.851809 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:38.851884 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:38.852143 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:04:38.852562 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:04:38.852652 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:38.852702 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 01:04:38.852749 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 01:04:38.858901 13302 raft_consensus.cc:1261] T bd018814d67f4de6b19a165350092680 P 8af14e4b74054f1692879fd3c74f7d1f [term 1 FOLLOWER]: Refusing update from remote peer d2f2fd82aba54a73958913d0bb98c81f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:04:38.858887 13418 raft_consensus.cc:1261] T bd018814d67f4de6b19a165350092680 P 6a5050c48dd648d188b9f762bb771a18 [term 1 FOLLOWER]: Refusing update from remote peer d2f2fd82aba54a73958913d0bb98c81f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:04:38.859570 13656 consensus_queue.cc:1035] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8af14e4b74054f1692879fd3c74f7d1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:04:38.859800 13670 consensus_queue.cc:1035] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a5050c48dd648d188b9f762bb771a18" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:04:38.874598 13149 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:04:38.874812 13149 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:04:38.877218 13149 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be707f8a7d50402ea7b63a44bffe01b7: Sending DeleteTablet for 3 replicas of tablet bd018814d67f4de6b19a165350092680
I20260929 01:04:38.878000 13398 tablet_service.cc:1514] Processing DeleteTablet for tablet bd018814d67f4de6b19a165350092680 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:04:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46190
I20260929 01:04:38.878036 13282 tablet_service.cc:1514] Processing DeleteTablet for tablet bd018814d67f4de6b19a165350092680 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:04:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54170
I20260929 01:04:38.878276 13550 tablet_service.cc:1514] Processing DeleteTablet for tablet bd018814d67f4de6b19a165350092680 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:04:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60248
I20260929 01:04:38.878530 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 13188
I20260929 01:04:38.878857 13684 tablet_replica.cc:330] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f: stopping tablet replica
I20260929 01:04:38.879231 13684 raft_consensus.cc:2229] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:04:38.880066 13684 raft_consensus.cc:2258] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:04:38.881633 13684 ts_tablet_manager.cc:1905] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:04:38.883019 13682 tablet_replica.cc:330] T bd018814d67f4de6b19a165350092680 P 6a5050c48dd648d188b9f762bb771a18: stopping tablet replica
I20260929 01:04:38.883339 13682 raft_consensus.cc:2229] T bd018814d67f4de6b19a165350092680 P 6a5050c48dd648d188b9f762bb771a18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:04:38.883647 13682 raft_consensus.cc:2258] T bd018814d67f4de6b19a165350092680 P 6a5050c48dd648d188b9f762bb771a18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:04:38.884496 13684 ts_tablet_manager.cc:1918] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:04:38.884595 13684 log.cc:1196] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/bd018814d67f4de6b19a165350092680
I20260929 01:04:38.884955 13684 ts_tablet_manager.cc:1939] T bd018814d67f4de6b19a165350092680 P d2f2fd82aba54a73958913d0bb98c81f: Deleting consensus metadata
I20260929 01:04:38.885133 13682 ts_tablet_manager.cc:1905] T bd018814d67f4de6b19a165350092680 P 6a5050c48dd648d188b9f762bb771a18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:04:38.886382 13135 catalog_manager.cc:4841] TS d2f2fd82aba54a73958913d0bb98c81f (127.0.98.3:45295): tablet bd018814d67f4de6b19a165350092680 (table test-table [id=e8738d9148ce4e199bd3bfe02c341bce]) successfully deleted
I20260929 01:04:38.887449 13682 ts_tablet_manager.cc:1918] T bd018814d67f4de6b19a165350092680 P 6a5050c48dd648d188b9f762bb771a18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:04:38.887558 13682 log.cc:1196] T bd018814d67f4de6b19a165350092680 P 6a5050c48dd648d188b9f762bb771a18: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790643720886468-392-0/minicluster-data/ts-1/wal/wals/bd018814d67f4de6b19a165350092680
I20260929 01:04:38.887859 13682 ts_tablet_manager.cc:1939] T bd018814d67f4de6b19a165350092680 P 6a5050c48dd648d188b9f762bb771a18: Deleting consensus metadata
I20260929 01:04:38.888978 13134 catalog_manager.cc:4841] TS 6a5050c48dd648d188b9f762bb771a18 (127.0.98.2:42031): tablet bd018814d67f4de6b19a165350092680 (table test-table [id=e8738d9148ce4e199bd3bfe02c341bce]) successfully deleted
W20260929 01:04:38.892771 13136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38763 (error 108)
I20260929 01:04:38.893589 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 13352
I20260929 01:04:38.893890 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:38.893588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38763 (local address 127.0.0.1:54174)
0929 01:04:38.893723 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:38.893794 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 01:04:38.894402 13136 catalog_manager.cc:4584] TS 8af14e4b74054f1692879fd3c74f7d1f (127.0.98.1:38763): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd018814d67f4de6b19a165350092680: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38763: connect: Connection refused (error 111)
I20260929 01:04:38.899381 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 13485
I20260929 01:04:38.904296 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 13117
I20260929 01:04:38.910579 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 01:04:39.245258 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 01:04:39.245357 392 mini_ranger.cc:66] Stopping Ranger...
I20260929 01:04:39.581140 392 mini_ranger.cc:68] Stopped Ranger
2026-09-29 01:04:39.581 UTC [12184] LOG: received smart shutdown request
2026-09-29 01:04:39.585 UTC [12184] LOG: background worker "logical replication launcher" (PID 12193) exited with exit code 1
2026-09-29 01:04:39.585 UTC [12188] LOG: shutting down
2026-09-29 01:04:39.746 UTC [12184] LOG: database system is shut down
2026-09-29T01:04:39Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48868 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:04:39 dist-test-slave-7zdl krb5kdc[13694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:04:39 dist-test-slave-7zdl krb5kdc[13694](info): set up 2 sockets
Sep 29 01:04:39 dist-test-slave-7zdl krb5kdc[13694](info): commencing operation
krb5kdc: starting...
W20260929 01:04:40.902932 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:04:40 dist-test-slave-7zdl krb5kdc[13694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:04:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:04:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:04:41.020942 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38835
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38835,127.0.98.61:43103,127.0.98.60:45935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:04:41.129792 13707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:41.130067 13707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:41.130136 13707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:41.134208 13707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:04:41.134316 13707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:41.134351 13707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:41.134392 13707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:04:41.134425 13707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:04:41.139729 13707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38835,127.0.98.61:43103,127.0.98.60:45935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:04:41.140812 13707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:41.141840 13707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:41.150364 13715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:41.150360 13713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:41.150362 13712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:41.150758 13707 server_base.cc:1034] running on GCE node
I20260929 01:04:41.151314 13707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:41.152382 13707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:41.153568 13707 hybrid_clock.cc:648] HybridClock initialized: now 1790643881153555 us; error 25 us; skew 500 ppm
Sep 29 01:04:41 dist-test-slave-7zdl krb5kdc[13694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:41.156279 13707 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:04:41.157157 13707 webserver.cc:460] Webserver started at http://127.0.98.62:45095/ using document root <none> and password file <none>
I20260929 01:04:41.157707 13707 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:41.157809 13707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:41.158026 13707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:04:41.160040 13707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "6b454cb34df24f95bb32b3cadac9cfb0"
format_stamp: "Formatted at 2026-09-29 01:04:41 on dist-test-slave-7zdl"
server_key: "e06d84ada9553985c293c00fe68863aa"
server_key_iv: "b76ce7d44e50d792ffd2a5826998f238"
server_key_version: "encryptionkey@0"
I20260929 01:04:41.160583 13707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "6b454cb34df24f95bb32b3cadac9cfb0"
format_stamp: "Formatted at 2026-09-29 01:04:41 on dist-test-slave-7zdl"
server_key: "e06d84ada9553985c293c00fe68863aa"
server_key_iv: "b76ce7d44e50d792ffd2a5826998f238"
server_key_version: "encryptionkey@0"
I20260929 01:04:41.164099 13707 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:04:41.166352 13722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:41.167491 13707 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:04:41.167632 13707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "6b454cb34df24f95bb32b3cadac9cfb0"
format_stamp: "Formatted at 2026-09-29 01:04:41 on dist-test-slave-7zdl"
server_key: "e06d84ada9553985c293c00fe68863aa"
server_key_iv: "b76ce7d44e50d792ffd2a5826998f238"
server_key_version: "encryptionkey@0"
I20260929 01:04:41.167735 13707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:41.178586 13707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:41.181237 13707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:41.181423 13707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:41.188540 13707 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38835
I20260929 01:04:41.188560 13774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38835 every 8 connection(s)
I20260929 01:04:41.189455 13707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:04:41.192126 13775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:41.195690 13775 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38835 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:04:41.196445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 13707
I20260929 01:04:41.196594 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:04:41.196856 392 external_mini_cluster.cc:1428] Setting key ca47ae87837f13afe8b9ea25cca24980
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 29 01:04:41 dist-test-slave-7zdl krb5kdc[13694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:04:41.213898 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.197951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38835 (local address 127.0.0.1:50370)
0929 01:04:41.198529 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.198583 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:41.199800 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:41.200470 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:41.200478 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:41.200982 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:41.201374 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:41.201394 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.202521 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.202527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:41.203837 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:41.203850 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.204464 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.204470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:41.204614 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:41.205189 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:41.205214 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:41.208989 (+ 3775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:41.210947 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:41.210959 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:41.210973 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:41.211298 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:41.211676 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:41.211682 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:41.211687 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:41.211841 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:41.212443 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:41.212453 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:41.212637 (+ 184us) client_negotiation.cc:769] Sending connection context
0929 01:04:41.212834 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 01:04:41.213251 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":81,"spinlock_wait_cycles":4992,"thread_start_us":146,"threads_started":1}
I20260929 01:04:41.213899 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.197950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50370 (local address 127.0.98.62:38835)
0929 01:04:41.198353 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:41.198372 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:41.198728 (+ 356us) server_negotiation.cc:407] Connection header received
0929 01:04:41.200089 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:41.200094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:41.200273 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:41.200415 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:41.201527 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.202389 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.203956 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.204303 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.204413 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:41.209137 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:41.209142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:41.209153 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:41.209175 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:41.210839 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.211434 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.211440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.211442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.211582 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.211958 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.211960 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.211961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.212319 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:41.212419 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:41.212896 (+ 477us) server_negotiation.cc:299] Negotiation successful
0929 01:04:41.213130 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":259,"thread_start_us":135,"threads_started":1}
I20260929 01:04:41.216269 13775 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:04:41.216980 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.216758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43103 (local address 127.0.0.1:45876)
0929 01:04:41.216863 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.216927 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 01:04:41.217180 13725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43103: connect: Connection refused (error 111)
I20260929 01:04:41.218024 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.217821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43103 (local address 127.0.0.1:45878)
0929 01:04:41.217929 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.217976 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 01:04:41.218154 13775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43103: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43103: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 01:04:41.223460 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43103
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38835,127.0.98.61:43103,127.0.98.60:45935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:04:41.267392 13775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 } attempt: 1
I20260929 01:04:41.268265 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.267970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43103 (local address 127.0.0.1:45880)
0929 01:04:41.268119 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.268188 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 01:04:41.268972 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.268757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43103 (local address 127.0.0.1:45886)
0929 01:04:41.268870 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.268919 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 01:04:41.269083 13775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43103: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43103: connect: Connection refused (error 111)
W20260929 01:04:41.330057 13782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:41.330363 13782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:41.330457 13782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:41.334460 13782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:04:41.334550 13782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:41.334589 13782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:41.334650 13782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:04:41.334692 13782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:04:41.337257 13775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 } attempt: 2
I20260929 01:04:41.338105 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.337774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43103 (local address 127.0.0.1:45890)
0929 01:04:41.337943 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.338015 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 01:04:41.338850 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.338608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43103 (local address 127.0.0.1:45902)
0929 01:04:41.338710 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.338759 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 01:04:41.338992 13775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43103: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43103: connect: Connection refused (error 111)
I20260929 01:04:41.340341 13782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38835,127.0.98.61:43103,127.0.98.60:45935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:04:41.341266 13782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:41.342195 13782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:41.350328 13790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:41.350328 13788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:41.350414 13787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:41.350479 13782 server_base.cc:1034] running on GCE node
I20260929 01:04:41.351200 13782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:41.352217 13782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:41.353374 13782 hybrid_clock.cc:648] HybridClock initialized: now 1790643881353355 us; error 31 us; skew 500 ppm
Sep 29 01:04:41 dist-test-slave-7zdl krb5kdc[13694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:41.356144 13782 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 01:04:41.356896 13782 webserver.cc:460] Webserver started at http://127.0.98.61:35671/ using document root <none> and password file <none>
I20260929 01:04:41.357455 13782 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:41.357517 13782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:41.357668 13782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:04:41.359611 13782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-1/data/instance:
uuid: "4168c6088f5241db99c906bc89995a99"
format_stamp: "Formatted at 2026-09-29 01:04:41 on dist-test-slave-7zdl"
server_key: "53ff67a1cae80da2b014c013b2bb5628"
server_key_iv: "a6c36f572a6a7eb4d9e1f7288d1f6f9d"
server_key_version: "encryptionkey@0"
I20260929 01:04:41.360095 13782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-1/wal/instance:
uuid: "4168c6088f5241db99c906bc89995a99"
format_stamp: "Formatted at 2026-09-29 01:04:41 on dist-test-slave-7zdl"
server_key: "53ff67a1cae80da2b014c013b2bb5628"
server_key_iv: "a6c36f572a6a7eb4d9e1f7288d1f6f9d"
server_key_version: "encryptionkey@0"
I20260929 01:04:41.363499 13782 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:04:41.365659 13797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:41.366947 13782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 01:04:41.367072 13782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-1/wal
uuid: "4168c6088f5241db99c906bc89995a99"
format_stamp: "Formatted at 2026-09-29 01:04:41 on dist-test-slave-7zdl"
server_key: "53ff67a1cae80da2b014c013b2bb5628"
server_key_iv: "a6c36f572a6a7eb4d9e1f7288d1f6f9d"
server_key_version: "encryptionkey@0"
I20260929 01:04:41.367168 13782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:41.388489 13782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:41.391407 13782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:41.391599 13782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:41.398394 13782 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43103
I20260929 01:04:41.398496 13849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43103 every 8 connection(s)
I20260929 01:04:41.399474 13782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-1/data/info.pb
I20260929 01:04:41.402146 13850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:41.405828 13850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38835 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:04:41.409054 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 13782
I20260929 01:04:41.409217 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-1/wal/instance
I20260929 01:04:41.409482 392 external_mini_cluster.cc:1428] Setting key 79d54d8be0c227889a3eea3998917c02
Sep 29 01:04:41 dist-test-slave-7zdl krb5kdc[13694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260929 01:04:41.419576 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.407763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35308 (local address 127.0.98.62:38835)
0929 01:04:41.407912 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:41.407915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:41.408695 (+ 780us) server_negotiation.cc:407] Connection header received
0929 01:04:41.409667 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:41.409670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:41.409735 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:41.409849 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:41.411024 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.411555 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.412832 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.413074 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.413187 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:41.415920 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:41.415923 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:41.415925 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:41.415951 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:41.417441 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.418009 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.418012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.418013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.418084 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.418481 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.418484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.418485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.418652 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:41.418728 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:41.419353 (+ 625us) server_negotiation.cc:299] Negotiation successful
0929 01:04:41.419457 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 01:04:41.420753 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.408102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38835 (local address 127.0.0.1:35308)
0929 01:04:41.408531 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.408572 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:41.409436 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:41.410014 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:41.410021 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:41.410358 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:41.410881 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:41.410896 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.411683 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.411686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:41.412700 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:41.412709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.413222 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.413226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:41.413399 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:41.414000 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:41.414016 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:41.415752 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:41.417669 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:41.417673 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:41.417684 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:41.417922 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:41.418198 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:41.418201 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:41.418203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:41.418406 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:41.418756 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:41.418762 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:41.419105 (+ 343us) client_negotiation.cc:769] Sending connection context
0929 01:04:41.419301 (+ 196us) client_negotiation.cc:240] Negotiation successful
0929 01:04:41.419662 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":102,"threads_started":1}
I20260929 01:04:41.422160 13850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:04:41 dist-test-slave-7zdl krb5kdc[13694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 01:04:41.430140 13775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 } attempt: 3
I20260929 01:04:41.433605 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.422724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43103 (local address 127.0.0.1:52900)
0929 01:04:41.422860 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.422874 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:41.422947 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:41.423679 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:41.423681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:41.423808 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:41.423888 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:41.423892 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.425382 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.425385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:41.425898 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:41.425906 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.426740 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.426742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:41.426921 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:41.427375 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:41.427386 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:41.429822 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:41.432012 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:41.432014 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:41.432016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:41.432260 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:41.432569 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:41.432572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:41.432574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:41.432681 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:41.433093 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:41.433096 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:41.433177 (+ 81us) client_negotiation.cc:769] Sending connection context
0929 01:04:41.433306 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 01:04:41.433421 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":4}
I20260929 01:04:41.434072 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.422924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52900 (local address 127.0.98.61:43103)
0929 01:04:41.423237 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:41.423249 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:41.423273 (+ 24us) server_negotiation.cc:407] Connection header received
0929 01:04:41.423332 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:41.423338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:41.423542 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:41.423692 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:41.423998 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.425266 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.426010 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.426624 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.426873 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:41.429938 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:41.429947 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:41.429997 (+ 50us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:41.430029 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:41.431899 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.432362 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.432370 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.432375 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.432466 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.432764 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.432767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.432769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.432986 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:41.433120 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:41.433382 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 01:04:41.433500 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":208,"thread_start_us":97,"threads_started":1}
Sep 29 01:04:41 dist-test-slave-7zdl krb5kdc[13694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 01:04:41.436045 13850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:04:41.436756 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.436511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45935 (local address 127.0.0.1:35398)
0929 01:04:41.436618 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.436695 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 01:04:41.436962 13799 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45935: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 01:04:41.438047 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.437812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45935 (local address 127.0.0.1:35402)
0929 01:04:41.437925 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.437983 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 01:04:41.438259 13850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45935: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45935: connect: Connection refused (error 111)
I20260929 01:04:41.438529 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45935
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38835,127.0.98.61:43103,127.0.98.60:45935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:04:41.440114 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.430557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43103 (local address 127.0.0.1:52914)
0929 01:04:41.430698 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.430714 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:41.430812 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:41.431258 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:41.431261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:41.431402 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:41.431495 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:41.431500 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.432447 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.432450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:41.433587 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:41.433597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.434144 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.434147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:41.434260 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:41.434681 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:41.434699 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:41.435934 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:41.438618 (+ 2684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:41.438621 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:41.438624 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:41.438900 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:41.439155 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:41.439161 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:41.439166 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:41.439265 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:41.439601 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:41.439607 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:41.439725 (+ 118us) client_negotiation.cc:769] Sending connection context
0929 01:04:41.439837 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 01:04:41.439988 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 01:04:41.440443 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.430649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52914 (local address 127.0.98.61:43103)
0929 01:04:41.431010 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:41.431016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:41.431026 (+ 10us) server_negotiation.cc:407] Connection header received
0929 01:04:41.431081 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:41.431084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:41.431127 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:41.431232 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:41.431606 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.432347 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.433720 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.434032 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.434145 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:41.436098 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:41.436101 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:41.436103 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:41.436122 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:41.438515 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.438995 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.438998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.438999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.439068 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.439366 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.439369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.439370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.439514 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:41.439582 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:41.439930 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 01:04:41.440054 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":219,"threads_started":1}
I20260929 01:04:41.441025 13775 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:04:41.441684 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.441453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45935 (local address 127.0.0.1:35404)
0929 01:04:41.441562 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.441613 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 01:04:41.442405 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.442185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45935 (local address 127.0.0.1:35408)
0929 01:04:41.442324 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.442363 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260929 01:04:41.442597 13775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45935: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45935: connect: Connection refused (error 111)
I20260929 01:04:41.473724 13775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } attempt: 1
I20260929 01:04:41.474540 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.474194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45935 (local address 127.0.0.1:35416)
0929 01:04:41.474333 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.474447 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 01:04:41.475267 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.475050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45935 (local address 127.0.0.1:35424)
0929 01:04:41.475159 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.475203 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 01:04:41.475426 13775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45935: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45935: connect: Connection refused (error 111)
I20260929 01:04:41.487515 13850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } attempt: 1
I20260929 01:04:41.488229 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.487974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45935 (local address 127.0.0.1:35428)
0929 01:04:41.488098 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.488161 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 01:04:41.488940 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.488732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45935 (local address 127.0.0.1:35436)
0929 01:04:41.488832 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.488877 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 01:04:41.489049 13850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45935: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45935: connect: Connection refused (error 111)
W20260929 01:04:41.547796 13858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:41.548060 13858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:41.548143 13858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 01:04:41.550573 13775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } attempt: 2
I20260929 01:04:41.551376 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.551094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45935 (local address 127.0.0.1:35450)
0929 01:04:41.551235 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.551297 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 01:04:41.552086 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.551867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45935 (local address 127.0.0.1:35462)
0929 01:04:41.551976 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.552029 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 01:04:41.552211 13775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45935: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45935: connect: Connection refused (error 111)
W20260929 01:04:41.552522 13858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:04:41.552610 13858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:41.552654 13858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:41.552690 13858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:04:41.552724 13858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:04:41.557188 13850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } attempt: 2
I20260929 01:04:41.557826 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.557584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45935 (local address 127.0.0.1:35464)
0929 01:04:41.557701 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.557760 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 01:04:41.558071 13858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38835,127.0.98.61:43103,127.0.98.60:45935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:04:41.558569 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.558362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45935 (local address 127.0.0.1:35480)
0929 01:04:41.558466 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.558512 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 01:04:41.558734 13850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45935: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45935: connect: Connection refused (error 111)
I20260929 01:04:41.559224 13858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:41.560194 13858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:41.568379 13866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:41.568452 13864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:41.568436 13863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:41.568543 13858 server_base.cc:1034] running on GCE node
I20260929 01:04:41.569381 13858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:41.570482 13858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:41.571638 13858 hybrid_clock.cc:648] HybridClock initialized: now 1790643881571617 us; error 30 us; skew 500 ppm
Sep 29 01:04:41 dist-test-slave-7zdl krb5kdc[13694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:41.574379 13858 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 01:04:41.575328 13858 webserver.cc:460] Webserver started at http://127.0.98.60:33123/ using document root <none> and password file <none>
I20260929 01:04:41.575868 13858 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:41.575956 13858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:41.576159 13858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:04:41.578119 13858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-2/data/instance:
uuid: "407d5a73a8174689b30c32aba0bcbc3e"
format_stamp: "Formatted at 2026-09-29 01:04:41 on dist-test-slave-7zdl"
server_key: "be026d938d9785819e9a4823a4592927"
server_key_iv: "9c15f6f7f84467c742196bfe9d1f5c66"
server_key_version: "encryptionkey@0"
I20260929 01:04:41.578675 13858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-2/wal/instance:
uuid: "407d5a73a8174689b30c32aba0bcbc3e"
format_stamp: "Formatted at 2026-09-29 01:04:41 on dist-test-slave-7zdl"
server_key: "be026d938d9785819e9a4823a4592927"
server_key_iv: "9c15f6f7f84467c742196bfe9d1f5c66"
server_key_version: "encryptionkey@0"
I20260929 01:04:41.582158 13858 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:04:41.584513 13873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:41.585675 13858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260929 01:04:41.585809 13858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-2/wal
uuid: "407d5a73a8174689b30c32aba0bcbc3e"
format_stamp: "Formatted at 2026-09-29 01:04:41 on dist-test-slave-7zdl"
server_key: "be026d938d9785819e9a4823a4592927"
server_key_iv: "9c15f6f7f84467c742196bfe9d1f5c66"
server_key_version: "encryptionkey@0"
I20260929 01:04:41.585911 13858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:41.609247 13858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:41.611948 13858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:41.612154 13858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:41.618897 13858 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45935
I20260929 01:04:41.618916 13925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45935 every 8 connection(s)
I20260929 01:04:41.619850 13858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-2/data/info.pb
I20260929 01:04:41.622458 13926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:41.625532 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 13858
I20260929 01:04:41.625658 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790643720886468-392-0/minicluster-data/master-2/wal/instance
I20260929 01:04:41.625970 392 external_mini_cluster.cc:1428] Setting key 942847b9a7bdafabb4b062098e73030d
I20260929 01:04:41.626240 13926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38835 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:04:41 dist-test-slave-7zdl krb5kdc[13694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 01:04:41 dist-test-slave-7zdl krb5kdc[13694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 01:04:41.638099 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.628251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35322 (local address 127.0.98.62:38835)
0929 01:04:41.628379 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:41.628384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:41.628678 (+ 294us) server_negotiation.cc:407] Connection header received
0929 01:04:41.629064 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:41.629067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:41.629131 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:41.629217 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:41.629987 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.630519 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.631598 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.631834 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.631949 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:41.634433 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:41.634439 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:41.634441 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:41.634469 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:41.636059 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.636565 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.636568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.636569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.636640 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.637028 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.637031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.637033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.637229 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:41.637346 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:41.637814 (+ 468us) server_negotiation.cc:299] Negotiation successful
0929 01:04:41.637933 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 01:04:41.638991 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.627520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35492 (local address 127.0.98.60:45935)
0929 01:04:41.627936 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:41.627953 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:41.627992 (+ 39us) server_negotiation.cc:407] Connection header received
0929 01:04:41.628818 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:41.628839 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:41.629040 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:41.629224 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:41.629997 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.630913 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.631727 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.632149 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.632266 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:41.634937 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:41.634945 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:41.634958 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:41.634989 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:41.636774 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.637156 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.637161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.637163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.637247 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.637512 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.637517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.637518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.637734 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:41.637923 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:41.638277 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 01:04:41.638426 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":260,"spinlock_wait_cycles":3840,"thread_start_us":123,"threads_started":1}
I20260929 01:04:41.638989 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.628157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38835 (local address 127.0.0.1:35322)
0929 01:04:41.628548 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.628588 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:41.628873 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:41.629267 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:41.629273 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:41.629599 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:41.629846 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:41.629859 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.630626 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.630631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:41.631436 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:41.631445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.631978 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.631982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:41.632179 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:41.632732 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:41.632756 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:41.634293 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:41.636161 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:41.636167 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:41.636177 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:41.636467 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:41.636728 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:41.636732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:41.636734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:41.636915 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:41.637357 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:41.637365 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:41.637591 (+ 226us) client_negotiation.cc:769] Sending connection context
0929 01:04:41.637770 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 01:04:41.638055 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":5,"spinlock_wait_cycles":3840,"thread_start_us":136,"threads_started":1}
I20260929 01:04:41.640165 13926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:04:41 dist-test-slave-7zdl krb5kdc[13694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 01:04:41 dist-test-slave-7zdl krb5kdc[13694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:04:41.648669 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.640831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.0.98.61:43103)
0929 01:04:41.640968 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:41.640972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:41.641052 (+ 80us) server_negotiation.cc:407] Connection header received
0929 01:04:41.641187 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:41.641190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:41.641244 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:41.641301 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:41.641718 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.642427 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.643312 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.643679 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.643766 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:41.645504 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:41.645508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:41.645510 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:41.645538 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:41.647043 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.647495 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.647497 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.647499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.647603 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.647885 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.647888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.647890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.648099 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:41.648199 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:41.648434 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 01:04:41.648560 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:04:41.648658 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.640641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43103 (local address 127.0.0.1:52924)
0929 01:04:41.640969 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.640985 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:41.641065 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:41.641413 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:41.641415 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:41.641538 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:41.641619 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:41.641623 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.642527 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.642530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:41.643199 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:41.643207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.643780 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.643783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:41.643876 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:41.644258 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:41.644268 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:41.645360 (+ 1092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:41.647163 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:41.647169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:41.647174 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:41.647407 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:41.647713 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:41.647715 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:41.647717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:41.647790 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:41.648210 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:41.648212 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:41.648280 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 01:04:41.648355 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 01:04:41.648483 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235}
I20260929 01:04:41.649439 13926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:04:41.649870 13850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } attempt: 3
I20260929 01:04:41.650152 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.641150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35332 (local address 127.0.98.62:38835)
0929 01:04:41.641250 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:41.641254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:41.641293 (+ 39us) server_negotiation.cc:407] Connection header received
0929 01:04:41.641430 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:41.641433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:41.641478 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:41.641556 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:41.642197 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.642768 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.643663 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.643965 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.644068 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:41.646676 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:41.646680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:41.646683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:41.646706 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:41.648385 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.648853 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.648856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.648858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.648938 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.649266 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.649269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.649270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.649481 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:41.649707 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:41.649878 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 01:04:41.649985 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 01:04:41.651351 13775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } attempt: 3
Sep 29 01:04:41 dist-test-slave-7zdl krb5kdc[13694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 01:04:41 dist-test-slave-7zdl krb5kdc[13694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 01:04:41 dist-test-slave-7zdl krb5kdc[13694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 01:04:41 dist-test-slave-7zdl krb5kdc[13694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 01:04:41.660808 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.649883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45935 (local address 127.0.0.1:35498)
0929 01:04:41.650082 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.650100 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:41.650154 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:41.651103 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:41.651105 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:41.651291 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:41.651376 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:41.651385 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.652368 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.652371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:41.653195 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:41.653213 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.653834 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.653837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:41.653933 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:41.654341 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:41.654355 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:41.655750 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:41.659318 (+ 3568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:41.659320 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:41.659323 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:41.659590 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:41.659854 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:41.659856 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:41.659858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:41.659967 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:41.660359 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:41.660361 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:41.660427 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 01:04:41.660517 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 01:04:41.660631 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":45}
I20260929 01:04:41.660965 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.651671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45935 (local address 127.0.0.1:35518)
0929 01:04:41.651897 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.651911 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:41.651981 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:41.652608 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:41.652614 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:41.652790 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:41.652872 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:41.652878 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.653996 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.654000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:41.654785 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:41.654815 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.655430 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.655434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:41.655514 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:41.655930 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:41.655949 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:41.657231 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:41.659499 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:41.659503 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:41.659506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:41.659734 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:41.660040 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:41.660043 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:41.660045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:41.660138 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:41.660539 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:41.660542 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:41.660628 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 01:04:41.660704 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 01:04:41.660816 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260929 01:04:41.661216 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.650718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35498 (local address 127.0.98.60:45935)
0929 01:04:41.650859 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:41.650862 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:41.650875 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:04:41.650905 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:41.650908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:41.650945 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:41.651014 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:41.651493 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.652251 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.653321 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.653660 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.653747 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:41.656983 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:41.656993 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:41.656996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:41.657022 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:41.659205 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.659681 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.659684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.659686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.659756 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.660048 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.660051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.660052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.660252 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:41.660329 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:41.660964 (+ 635us) server_negotiation.cc:299] Negotiation successful
0929 01:04:41.661066 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 01:04:41.661500 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.652116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35518 (local address 127.0.98.60:45935)
0929 01:04:41.652385 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:41.652393 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:41.652406 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:04:41.652435 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:41.652438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:41.652476 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:41.652974 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:41.653066 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.653876 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.654943 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.655295 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.655717 (+ 422us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:41.657379 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:41.657383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:41.657385 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:41.657409 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:41.659385 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.659844 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.659848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.659849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.659942 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.660242 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.660245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.660246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.660431 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:41.661197 (+ 766us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:41.661268 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 01:04:41.661380 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":205,"thread_start_us":102,"threads_started":1}
I20260929 01:04:41.661895 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.651272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52936 (local address 127.0.98.61:43103)
0929 01:04:41.651366 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:41.651370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:41.651580 (+ 210us) server_negotiation.cc:407] Connection header received
0929 01:04:41.651741 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:41.651744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:41.651782 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:41.651827 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:41.652573 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.653282 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.654560 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.655213 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.655298 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:41.658431 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:41.658435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:41.658438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:41.658464 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:41.660035 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.660517 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.660523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.660526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.660598 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.660902 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.660905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.660908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.661119 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:41.661279 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:41.661637 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 01:04:41.661754 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 01:04:41.663264 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.650301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45935 (local address 127.0.0.1:35504)
0929 01:04:41.650451 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:41.650464 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:41.650536 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:41.651385 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:41.651389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:41.651560 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:41.651661 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:41.651668 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.653213 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.653217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:41.654017 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:41.654026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.654621 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.654624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:41.654701 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:41.655147 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:41.655160 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:41.656614 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:41.660262 (+ 3648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:41.660265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:41.660267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:41.660487 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:41.660772 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:41.660775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:41.660776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:41.660859 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:41.662771 (+ 1912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:41.662773 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:41.662897 (+ 124us) client_negotiation.cc:769] Sending connection context
0929 01:04:41.662978 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 01:04:41.663092 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 01:04:41.664047 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.650739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35504 (local address 127.0.98.60:45935)
0929 01:04:41.651128 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:41.651138 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:41.651150 (+ 12us) server_negotiation.cc:407] Connection header received
0929 01:04:41.651192 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:41.651195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:41.651234 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:41.651766 (+ 532us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:41.651870 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.653095 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.654141 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.654502 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.654922 (+ 420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:41.656764 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:41.656770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:41.656776 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:41.656807 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:41.660138 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.660597 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.660599 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.660602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.660674 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.662454 (+ 1780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.662457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.662458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.662657 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:41.663706 (+ 1049us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:41.663780 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 01:04:41.663881 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":309,"thread_start_us":172,"threads_started":1}
I20260929 01:04:41.664613 13775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0: Bootstrap starting.
I20260929 01:04:41.664613 13926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 407d5a73a8174689b30c32aba0bcbc3e: Bootstrap starting.
I20260929 01:04:41.665351 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:04:41.667016 13850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4168c6088f5241db99c906bc89995a99: Bootstrap starting.
I20260929 01:04:41.667431 13775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:41.667431 13926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 407d5a73a8174689b30c32aba0bcbc3e: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:41.668308 13775 log.cc:824] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:41.668308 13926 log.cc:824] T 00000000000000000000000000000000 P 407d5a73a8174689b30c32aba0bcbc3e: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:41.669567 13850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4168c6088f5241db99c906bc89995a99: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:41.670234 13850 log.cc:824] T 00000000000000000000000000000000 P 4168c6088f5241db99c906bc89995a99: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:41.672096 13926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 407d5a73a8174689b30c32aba0bcbc3e: No bootstrap required, opened a new log
I20260929 01:04:41.675046 13850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4168c6088f5241db99c906bc89995a99: No bootstrap required, opened a new log
I20260929 01:04:41.677080 13926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 407d5a73a8174689b30c32aba0bcbc3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38835 } } peers { permanent_uuid: "4168c6088f5241db99c906bc89995a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 } } peers { permanent_uuid: "407d5a73a8174689b30c32aba0bcbc3e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } }
I20260929 01:04:41.677362 13926 raft_consensus.cc:374] T 00000000000000000000000000000000 P 407d5a73a8174689b30c32aba0bcbc3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:41.677430 13926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 407d5a73a8174689b30c32aba0bcbc3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 407d5a73a8174689b30c32aba0bcbc3e, State: Initialized, Role: FOLLOWER
I20260929 01:04:41.677944 13926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 407d5a73a8174689b30c32aba0bcbc3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38835 } } peers { permanent_uuid: "4168c6088f5241db99c906bc89995a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 } } peers { permanent_uuid: "407d5a73a8174689b30c32aba0bcbc3e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } }
I20260929 01:04:41.678131 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.668473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52942 (local address 127.0.98.61:43103)
0929 01:04:41.668560 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:41.668564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:41.668977 (+ 413us) server_negotiation.cc:407] Connection header received
0929 01:04:41.669128 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:41.669131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:41.669169 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:41.669218 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:41.670193 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.670965 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.672469 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.672759 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.672846 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:41.674435 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:41.674439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:41.674442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:41.674466 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:41.676310 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.676802 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.676806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.676808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.676895 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.677190 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.677193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.677195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.677410 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:41.677538 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:41.677867 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 01:04:41.677978 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 01:04:41.678896 13850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4168c6088f5241db99c906bc89995a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38835 } } peers { permanent_uuid: "4168c6088f5241db99c906bc89995a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 } } peers { permanent_uuid: "407d5a73a8174689b30c32aba0bcbc3e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } }
I20260929 01:04:41.679105 13850 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4168c6088f5241db99c906bc89995a99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:41.679152 13850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4168c6088f5241db99c906bc89995a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4168c6088f5241db99c906bc89995a99, State: Initialized, Role: FOLLOWER
I20260929 01:04:41.679433 13926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 407d5a73a8174689b30c32aba0bcbc3e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:04:41.680224 13850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4168c6088f5241db99c906bc89995a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38835 } } peers { permanent_uuid: "4168c6088f5241db99c906bc89995a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 } } peers { permanent_uuid: "407d5a73a8174689b30c32aba0bcbc3e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } }
I20260929 01:04:41.681989 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4168c6088f5241db99c906bc89995a99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38835 } } peers { permanent_uuid: "4168c6088f5241db99c906bc89995a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 } } peers { permanent_uuid: "407d5a73a8174689b30c32aba0bcbc3e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } } }
I20260929 01:04:41.682137 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4168c6088f5241db99c906bc89995a99 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:04:41.684194 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.667881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35340 (local address 127.0.98.62:38835)
0929 01:04:41.667982 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:41.667986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:41.668918 (+ 932us) server_negotiation.cc:407] Connection header received
0929 01:04:41.669105 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:41.669111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:41.669151 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:41.669202 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:41.670622 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.671366 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.672333 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.672558 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.672665 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:41.680034 (+ 7369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:41.680041 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:41.680048 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:41.680081 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:41.681855 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.682697 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.682700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.682702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.682789 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.683131 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.683140 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.683144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.683414 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:41.683524 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:41.683908 (+ 384us) server_negotiation.cc:299] Negotiation successful
0929 01:04:41.684021 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 01:04:41.684711 13955 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 407d5a73a8174689b30c32aba0bcbc3e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:04:41.684778 13955 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:04:41.685500 13850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4168c6088f5241db99c906bc89995a99 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:04:41.683138 13858 master_runner.cc:186] Error getting master registration for 127.0.98.62:38835: OK, instance_id { permanent_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" instance_seqno: 1790643881186438 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:04:41.686384 13858 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:04:41.686674 13775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0: No bootstrap required, opened a new log
I20260929 01:04:41.688758 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:41.669057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35522 (local address 127.0.98.60:45935)
0929 01:04:41.669145 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:41.669149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:41.678124 (+ 8975us) server_negotiation.cc:407] Connection header received
0929 01:04:41.678412 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:41.678416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:41.678460 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:41.678506 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:41.679448 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.680300 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.682227 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:41.682581 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:41.684149 (+ 1568us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:41.684451 (+ 302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:41.684456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:41.684459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:41.684487 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:41.686491 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.687099 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.687102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.687104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.687193 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:41.687493 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:41.687496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:41.687498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:41.687739 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:41.688394 (+ 655us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:41.688487 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 01:04:41.688592 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 01:04:41.690560 13775 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38835 } } peers { permanent_uuid: "4168c6088f5241db99c906bc89995a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 } } peers { permanent_uuid: "407d5a73a8174689b30c32aba0bcbc3e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } }
I20260929 01:04:41.690762 13775 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:41.690838 13775 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b454cb34df24f95bb32b3cadac9cfb0, State: Initialized, Role: FOLLOWER
W20260929 01:04:41.690959 13967 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4168c6088f5241db99c906bc89995a99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:04:41.691026 13967 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:04:41.691299 13775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38835 } } peers { permanent_uuid: "4168c6088f5241db99c906bc89995a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 } } peers { permanent_uuid: "407d5a73a8174689b30c32aba0bcbc3e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } }
I20260929 01:04:41.689296 13782 master_runner.cc:186] Error getting master registration for 127.0.98.62:38835: OK, instance_id { permanent_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" instance_seqno: 1790643881186438 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:04:41.692406 13782 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:04:41.692476 13775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:04:41.694662 13943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 407d5a73a8174689b30c32aba0bcbc3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38835 } } peers { permanent_uuid: "4168c6088f5241db99c906bc89995a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 } } peers { permanent_uuid: "407d5a73a8174689b30c32aba0bcbc3e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } } }
I20260929 01:04:41.694779 13943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 407d5a73a8174689b30c32aba0bcbc3e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:04:41.694947 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38835 } } peers { permanent_uuid: "4168c6088f5241db99c906bc89995a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 } } peers { permanent_uuid: "407d5a73a8174689b30c32aba0bcbc3e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } } }
I20260929 01:04:41.695065 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:04:41.699159 13980 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:04:41.699244 13980 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:04:41.701622 13707 master_runner.cc:263] No leader master found from master 6b454cb34df24f95bb32b3cadac9cfb0
I20260929 01:04:41.701720 13707 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:04:41.797320 13969 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:04:41.797529 13969 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38835 } } peers { permanent_uuid: "4168c6088f5241db99c906bc89995a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 } } peers { permanent_uuid: "407d5a73a8174689b30c32aba0bcbc3e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } }
I20260929 01:04:41.798666 13969 leader_election.cc:288] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4168c6088f5241db99c906bc89995a99 (127.0.98.61:43103), 407d5a73a8174689b30c32aba0bcbc3e (127.0.98.60:45935)
I20260929 01:04:41.799419 13824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4168c6088f5241db99c906bc89995a99" is_pre_election: true
I20260929 01:04:41.799769 13824 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4168c6088f5241db99c906bc89995a99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b454cb34df24f95bb32b3cadac9cfb0 in term 0.
I20260929 01:04:41.800315 13725 leader_election.cc:302] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4168c6088f5241db99c906bc89995a99, 6b454cb34df24f95bb32b3cadac9cfb0; no voters:
I20260929 01:04:41.801016 13899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "407d5a73a8174689b30c32aba0bcbc3e" is_pre_election: true
I20260929 01:04:41.801317 13899 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 407d5a73a8174689b30c32aba0bcbc3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b454cb34df24f95bb32b3cadac9cfb0 in term 0.
I20260929 01:04:41.801616 13969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:04:41.801698 13969 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:04:41.801743 13969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:41.803489 13969 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38835 } } peers { permanent_uuid: "4168c6088f5241db99c906bc89995a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 } } peers { permanent_uuid: "407d5a73a8174689b30c32aba0bcbc3e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } }
I20260929 01:04:41.803812 13969 leader_election.cc:288] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [CANDIDATE]: Term 1 election: Requested vote from peers 4168c6088f5241db99c906bc89995a99 (127.0.98.61:43103), 407d5a73a8174689b30c32aba0bcbc3e (127.0.98.60:45935)
I20260929 01:04:41.804327 13824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4168c6088f5241db99c906bc89995a99"
I20260929 01:04:41.804445 13824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4168c6088f5241db99c906bc89995a99 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:41.805720 13824 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4168c6088f5241db99c906bc89995a99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b454cb34df24f95bb32b3cadac9cfb0 in term 1.
I20260929 01:04:41.806196 13899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "407d5a73a8174689b30c32aba0bcbc3e"
I20260929 01:04:41.806332 13899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 407d5a73a8174689b30c32aba0bcbc3e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:41.807622 13899 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 407d5a73a8174689b30c32aba0bcbc3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b454cb34df24f95bb32b3cadac9cfb0 in term 1.
I20260929 01:04:41.807894 13725 leader_election.cc:302] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4168c6088f5241db99c906bc89995a99, 6b454cb34df24f95bb32b3cadac9cfb0; no voters:
I20260929 01:04:41.808359 13969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:04:41.808597 13969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [term 1 LEADER]: Becoming Leader. State: Replica: 6b454cb34df24f95bb32b3cadac9cfb0, State: Running, Role: LEADER
I20260929 01:04:41.808913 13969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38835 } } peers { permanent_uuid: "4168c6088f5241db99c906bc89995a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 } } peers { permanent_uuid: "407d5a73a8174689b30c32aba0bcbc3e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } }
I20260929 01:04:41.810693 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b454cb34df24f95bb32b3cadac9cfb0. Latest consensus state: current_term: 1 leader_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38835 } } peers { permanent_uuid: "4168c6088f5241db99c906bc89995a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 } } peers { permanent_uuid: "407d5a73a8174689b30c32aba0bcbc3e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } } }
I20260929 01:04:41.813375 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:41.815265 13984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:04:41.816283 13984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:04:41.821892 13824 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4168c6088f5241db99c906bc89995a99 [term 1 FOLLOWER]: Refusing update from remote peer 6b454cb34df24f95bb32b3cadac9cfb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:04:41.822751 13899 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 407d5a73a8174689b30c32aba0bcbc3e [term 1 FOLLOWER]: Refusing update from remote peer 6b454cb34df24f95bb32b3cadac9cfb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:04:41.823594 13982 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "407d5a73a8174689b30c32aba0bcbc3e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:04:41.823963 13969 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4168c6088f5241db99c906bc89995a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:04:41.827685 13943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 407d5a73a8174689b30c32aba0bcbc3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b454cb34df24f95bb32b3cadac9cfb0. Latest consensus state: current_term: 1 leader_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38835 } } peers { permanent_uuid: "4168c6088f5241db99c906bc89995a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 } } peers { permanent_uuid: "407d5a73a8174689b30c32aba0bcbc3e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } } }
I20260929 01:04:41.827695 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4168c6088f5241db99c906bc89995a99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b454cb34df24f95bb32b3cadac9cfb0. Latest consensus state: current_term: 1 leader_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38835 } } peers { permanent_uuid: "4168c6088f5241db99c906bc89995a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 } } peers { permanent_uuid: "407d5a73a8174689b30c32aba0bcbc3e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } } }
I20260929 01:04:41.827806 13943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 407d5a73a8174689b30c32aba0bcbc3e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:04:41.827812 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4168c6088f5241db99c906bc89995a99 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:04:41.833238 13985 mvcc.cc:205] Tried to move back new op lower bound from 7334477339934466048 to 7334477339892822016. Current Snapshot: MvccSnapshot[applied={T|T < 7334477339934466048}]
I20260929 01:04:41.833664 13982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38835 } } peers { permanent_uuid: "4168c6088f5241db99c906bc89995a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 } } peers { permanent_uuid: "407d5a73a8174689b30c32aba0bcbc3e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } } }
I20260929 01:04:41.833799 13982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:41.834600 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4168c6088f5241db99c906bc89995a99 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38835 } } peers { permanent_uuid: "4168c6088f5241db99c906bc89995a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 } } peers { permanent_uuid: "407d5a73a8174689b30c32aba0bcbc3e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } } }
I20260929 01:04:41.834749 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4168c6088f5241db99c906bc89995a99 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:04:41.837210 13988 mvcc.cc:205] Tried to move back new op lower bound from 7334477339934466048 to 7334477339892822016. Current Snapshot: MvccSnapshot[applied={T|T < 7334477339934466048}]
I20260929 01:04:41.840828 13982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38835 } } peers { permanent_uuid: "4168c6088f5241db99c906bc89995a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 } } peers { permanent_uuid: "407d5a73a8174689b30c32aba0bcbc3e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } } }
I20260929 01:04:41.840940 13982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0 [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:41.841818 13943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 407d5a73a8174689b30c32aba0bcbc3e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b454cb34df24f95bb32b3cadac9cfb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38835 } } peers { permanent_uuid: "4168c6088f5241db99c906bc89995a99" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43103 } } peers { permanent_uuid: "407d5a73a8174689b30c32aba0bcbc3e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45935 } } }
I20260929 01:04:41.841945 13943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 407d5a73a8174689b30c32aba0bcbc3e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:04:41.842352 13987 mvcc.cc:205] Tried to move back new op lower bound from 7334477339934466048 to 7334477339892822016. Current Snapshot: MvccSnapshot[applied={T|T < 7334477339934466048}]
I20260929 01:04:41.842833 13984 catalog_manager.cc:1347] Generated new cluster ID: 7614c818e385476ab44801fabb1ca589
I20260929 01:04:41.842896 13984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:04:41.872383 13984 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:04:41.873602 13984 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:04:41.883433 13984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b454cb34df24f95bb32b3cadac9cfb0: Generated new TSK 0
I20260929 01:04:41.884222 13984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:04:42.686064 13955 catalog_manager.cc:1259] Loaded cluster ID: 7614c818e385476ab44801fabb1ca589
I20260929 01:04:42.686190 13955 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 407d5a73a8174689b30c32aba0bcbc3e: loading cluster ID for follower catalog manager: success
I20260929 01:04:42.688057 13955 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 407d5a73a8174689b30c32aba0bcbc3e: acquiring CA information for follower catalog manager: success
I20260929 01:04:42.689174 13955 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 407d5a73a8174689b30c32aba0bcbc3e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:04:42.692042 13967 catalog_manager.cc:1259] Loaded cluster ID: 7614c818e385476ab44801fabb1ca589
I20260929 01:04:42.692098 13967 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4168c6088f5241db99c906bc89995a99: loading cluster ID for follower catalog manager: success
I20260929 01:04:42.693650 13967 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4168c6088f5241db99c906bc89995a99: acquiring CA information for follower catalog manager: success
I20260929 01:04:42.694702 13967 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4168c6088f5241db99c906bc89995a99: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:04:42.941748 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 13707
I20260929 01:04:42.948199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 13782
I20260929 01:04:42.953677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 13858
2026-09-29T01:04:42Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3084 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:04:43 dist-test-slave-7zdl krb5kdc[14000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:04:43 dist-test-slave-7zdl krb5kdc[14000](info): set up 2 sockets
Sep 29 01:04:43 dist-test-slave-7zdl krb5kdc[14000](info): commencing operation
krb5kdc: starting...
W20260929 01:04:43.990898 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:04:44 dist-test-slave-7zdl krb5kdc[14000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:04:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:04:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:04:44.108076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35921
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:04:44.216701 14013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:44.216987 14013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:44.217068 14013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:44.221091 14013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:04:44.221175 14013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:44.221231 14013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:44.221288 14013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:04:44.221328 14013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:04:44.226473 14013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:04:44.227571 14013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:44.228582 14013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:44.237041 14021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:44.237078 14013 server_base.cc:1034] running on GCE node
W20260929 01:04:44.237088 14018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:44.237161 14019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:44.237854 14013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:44.239033 14013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:44.240211 14013 hybrid_clock.cc:648] HybridClock initialized: now 1790643884240179 us; error 44 us; skew 500 ppm
Sep 29 01:04:44 dist-test-slave-7zdl krb5kdc[14000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:44.242915 14013 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:04:44.243825 14013 webserver.cc:460] Webserver started at http://127.0.98.62:42147/ using document root <none> and password file <none>
I20260929 01:04:44.244374 14013 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:44.244462 14013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:44.244688 14013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:04:44.246634 14013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "556b1d78a743448db2d4ebaaa1ab20eb"
format_stamp: "Formatted at 2026-09-29 01:04:44 on dist-test-slave-7zdl"
server_key: "b66f12eb0fb3655488b34cbc0b54f653"
server_key_iv: "e418fb1ea87d01f38a44640b4b5f2a4a"
server_key_version: "encryptionkey@0"
I20260929 01:04:44.247242 14013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "556b1d78a743448db2d4ebaaa1ab20eb"
format_stamp: "Formatted at 2026-09-29 01:04:44 on dist-test-slave-7zdl"
server_key: "b66f12eb0fb3655488b34cbc0b54f653"
server_key_iv: "e418fb1ea87d01f38a44640b4b5f2a4a"
server_key_version: "encryptionkey@0"
I20260929 01:04:44.250738 14013 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:04:44.253511 14028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:44.254683 14013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:04:44.254858 14013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "556b1d78a743448db2d4ebaaa1ab20eb"
format_stamp: "Formatted at 2026-09-29 01:04:44 on dist-test-slave-7zdl"
server_key: "b66f12eb0fb3655488b34cbc0b54f653"
server_key_iv: "e418fb1ea87d01f38a44640b4b5f2a4a"
server_key_version: "encryptionkey@0"
I20260929 01:04:44.254984 14013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:44.266772 14013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:44.269457 14013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:44.269657 14013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:44.276492 14013 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35921
I20260929 01:04:44.276551 14080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35921 every 8 connection(s)
I20260929 01:04:44.277451 14013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:04:44.280169 14081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:44.284014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 14013
I20260929 01:04:44.284121 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:04:44.284425 392 external_mini_cluster.cc:1428] Setting key 9c4538c125994f7ea2996696217edc79
I20260929 01:04:44.286114 14081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 556b1d78a743448db2d4ebaaa1ab20eb: Bootstrap starting.
I20260929 01:04:44.288578 14081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 556b1d78a743448db2d4ebaaa1ab20eb: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:44.289283 14081 log.cc:824] T 00000000000000000000000000000000 P 556b1d78a743448db2d4ebaaa1ab20eb: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:44.291328 14081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 556b1d78a743448db2d4ebaaa1ab20eb: No bootstrap required, opened a new log
Sep 29 01:04:44 dist-test-slave-7zdl krb5kdc[14000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:04:44.295243 14081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 556b1d78a743448db2d4ebaaa1ab20eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556b1d78a743448db2d4ebaaa1ab20eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35921 } }
I20260929 01:04:44.295475 14081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 556b1d78a743448db2d4ebaaa1ab20eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:44.295563 14081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 556b1d78a743448db2d4ebaaa1ab20eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 556b1d78a743448db2d4ebaaa1ab20eb, State: Initialized, Role: FOLLOWER
I20260929 01:04:44.296027 14081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 556b1d78a743448db2d4ebaaa1ab20eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556b1d78a743448db2d4ebaaa1ab20eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35921 } }
I20260929 01:04:44.296175 14081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 556b1d78a743448db2d4ebaaa1ab20eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:04:44.296262 14081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 556b1d78a743448db2d4ebaaa1ab20eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:04:44.296360 14081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 556b1d78a743448db2d4ebaaa1ab20eb [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:44.297202 14081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 556b1d78a743448db2d4ebaaa1ab20eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556b1d78a743448db2d4ebaaa1ab20eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35921 } }
I20260929 01:04:44.297554 14081 leader_election.cc:302] T 00000000000000000000000000000000 P 556b1d78a743448db2d4ebaaa1ab20eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 556b1d78a743448db2d4ebaaa1ab20eb; no voters:
I20260929 01:04:44.297896 14081 leader_election.cc:288] T 00000000000000000000000000000000 P 556b1d78a743448db2d4ebaaa1ab20eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:04:44.298032 14086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 556b1d78a743448db2d4ebaaa1ab20eb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:04:44.298214 14086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 556b1d78a743448db2d4ebaaa1ab20eb [term 1 LEADER]: Becoming Leader. State: Replica: 556b1d78a743448db2d4ebaaa1ab20eb, State: Running, Role: LEADER
I20260929 01:04:44.298555 14086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 556b1d78a743448db2d4ebaaa1ab20eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556b1d78a743448db2d4ebaaa1ab20eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35921 } }
I20260929 01:04:44.299050 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:44.285738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57324 (local address 127.0.98.62:35921)
0929 01:04:44.286135 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:44.286151 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:44.286186 (+ 35us) server_negotiation.cc:407] Connection header received
0929 01:04:44.286896 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:44.286913 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:44.287235 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:44.287509 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:44.288410 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:44.289512 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:44.290629 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:44.291166 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:44.291305 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:44.293973 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:44.293978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:44.293990 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:44.294015 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:44.295882 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:44.296349 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:44.296354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:44.296372 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:44.296509 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:44.296802 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:44.296807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:44.296809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:44.297185 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:44.297383 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:44.297681 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 01:04:44.297999 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":134,"threads_started":1}
I20260929 01:04:44.299170 14081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 556b1d78a743448db2d4ebaaa1ab20eb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:04:44.300199 14088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 556b1d78a743448db2d4ebaaa1ab20eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "556b1d78a743448db2d4ebaaa1ab20eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556b1d78a743448db2d4ebaaa1ab20eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35921 } } }
I20260929 01:04:44.300324 14088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 556b1d78a743448db2d4ebaaa1ab20eb [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:44.300736 14095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:04:44.301796 14087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 556b1d78a743448db2d4ebaaa1ab20eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 556b1d78a743448db2d4ebaaa1ab20eb. Latest consensus state: current_term: 1 leader_uuid: "556b1d78a743448db2d4ebaaa1ab20eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556b1d78a743448db2d4ebaaa1ab20eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35921 } } }
I20260929 01:04:44.301895 14087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 556b1d78a743448db2d4ebaaa1ab20eb [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:44.302559 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:04:44.303877 14095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:04:44.309286 14095 catalog_manager.cc:1347] Generated new cluster ID: 2305dca238b74722bfd32429500f8de1
I20260929 01:04:44.309350 14095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:04:44.332472 14095 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:04:44.333808 14095 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:04:44.340862 14095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 556b1d78a743448db2d4ebaaa1ab20eb: Generated new TSK 0
I20260929 01:04:44.341621 14095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:04:44.358688 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 14013
2026-09-29T01:04:44Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1402 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:04:44 dist-test-slave-7zdl krb5kdc[14108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:04:44 dist-test-slave-7zdl krb5kdc[14108](info): set up 2 sockets
Sep 29 01:04:44 dist-test-slave-7zdl krb5kdc[14108](info): commencing operation
krb5kdc: starting...
W20260929 01:04:45.383117 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:04:45 dist-test-slave-7zdl krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:04:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:04:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:04:45.501343 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41481
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:04:45.607542 14121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:45.607826 14121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:45.607905 14121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:45.611857 14121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:04:45.611931 14121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:45.611967 14121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:45.612025 14121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:04:45.612066 14121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:04:45.617239 14121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:04:45.618295 14121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:45.619365 14121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:45.627635 14126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:45.627652 14129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:45.627684 14127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:45.627731 14121 server_base.cc:1034] running on GCE node
I20260929 01:04:45.628414 14121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:45.629504 14121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:45.630671 14121 hybrid_clock.cc:648] HybridClock initialized: now 1790643885630657 us; error 26 us; skew 500 ppm
Sep 29 01:04:45 dist-test-slave-7zdl krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:45.640790 14121 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:04:45.641728 14121 webserver.cc:460] Webserver started at http://127.0.98.62:41195/ using document root <none> and password file <none>
I20260929 01:04:45.642258 14121 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:45.642316 14121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:45.642480 14121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:04:45.644557 14121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "72a783dac216456a9482b052f226ff5e"
format_stamp: "Formatted at 2026-09-29 01:04:45 on dist-test-slave-7zdl"
server_key: "13e1231a8fe278fc6d740d0415c80d3f"
server_key_iv: "8ce6e14e470b131a752cf61f97d0b3f4"
server_key_version: "encryptionkey@0"
I20260929 01:04:45.645056 14121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "72a783dac216456a9482b052f226ff5e"
format_stamp: "Formatted at 2026-09-29 01:04:45 on dist-test-slave-7zdl"
server_key: "13e1231a8fe278fc6d740d0415c80d3f"
server_key_iv: "8ce6e14e470b131a752cf61f97d0b3f4"
server_key_version: "encryptionkey@0"
I20260929 01:04:45.648573 14121 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:04:45.650749 14136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:45.651968 14121 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:04:45.652091 14121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "72a783dac216456a9482b052f226ff5e"
format_stamp: "Formatted at 2026-09-29 01:04:45 on dist-test-slave-7zdl"
server_key: "13e1231a8fe278fc6d740d0415c80d3f"
server_key_iv: "8ce6e14e470b131a752cf61f97d0b3f4"
server_key_version: "encryptionkey@0"
I20260929 01:04:45.652197 14121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:45.695338 14121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:45.698244 14121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:45.698450 14121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:45.705374 14121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41481
I20260929 01:04:45.705739 14188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41481 every 8 connection(s)
I20260929 01:04:45.706637 14121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:04:45.707226 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 14121
I20260929 01:04:45.707377 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:04:45.707752 392 external_mini_cluster.cc:1428] Setting key 39cb0930a5c852d6475e272e3fe22715
I20260929 01:04:45.711567 14189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:45.717648 14189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e: Bootstrap starting.
Sep 29 01:04:45 dist-test-slave-7zdl krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:04:45.720309 14189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:45.721103 14189 log.cc:824] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:45.723253 14189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e: No bootstrap required, opened a new log
I20260929 01:04:45.724076 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:45.709284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43758 (local address 127.0.98.62:41481)
0929 01:04:45.710406 (+ 1122us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:45.710426 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:45.710497 (+ 71us) server_negotiation.cc:407] Connection header received
0929 01:04:45.711648 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:45.711676 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:45.712194 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:45.712604 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:45.714177 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:45.715288 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:45.716149 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:45.716641 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:45.716838 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:45.719602 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:45.719607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:45.719620 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:45.719647 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:45.721388 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:45.721828 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:45.721834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:45.721846 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:45.721959 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:45.722249 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:45.722252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:45.722253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:45.722608 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:45.722760 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:45.723063 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 01:04:45.723367 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":911,"thread_start_us":145,"threads_started":1}
I20260929 01:04:45.726588 14189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a783dac216456a9482b052f226ff5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41481 } }
I20260929 01:04:45.726840 14189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:45.726907 14189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72a783dac216456a9482b052f226ff5e, State: Initialized, Role: FOLLOWER
I20260929 01:04:45.727252 14189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a783dac216456a9482b052f226ff5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41481 } }
I20260929 01:04:45.727362 14189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:04:45.727403 14189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:04:45.727469 14189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:45.728245 14189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a783dac216456a9482b052f226ff5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41481 } }
I20260929 01:04:45.728516 14189 leader_election.cc:302] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72a783dac216456a9482b052f226ff5e; no voters:
I20260929 01:04:45.728799 14189 leader_election.cc:288] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:04:45.728888 14194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:04:45.729094 14194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [term 1 LEADER]: Becoming Leader. State: Replica: 72a783dac216456a9482b052f226ff5e, State: Running, Role: LEADER
I20260929 01:04:45.729434 14194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a783dac216456a9482b052f226ff5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41481 } }
I20260929 01:04:45.729925 14189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:04:45.731348 14196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72a783dac216456a9482b052f226ff5e. Latest consensus state: current_term: 1 leader_uuid: "72a783dac216456a9482b052f226ff5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a783dac216456a9482b052f226ff5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41481 } } }
I20260929 01:04:45.731741 14196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:45.732270 14195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72a783dac216456a9482b052f226ff5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a783dac216456a9482b052f226ff5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41481 } } }
I20260929 01:04:45.732331 14209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:04:45.732367 14195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:45.735271 14209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:04:45.740270 14209 catalog_manager.cc:1347] Generated new cluster ID: 8f4804fedb6848e99acedd943c9d443d
I20260929 01:04:45.740334 14209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:04:45.752005 14209 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:04:45.753165 14209 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:04:45.762606 14209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e: Generated new TSK 0
I20260929 01:04:45.763304 14209 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 01:04:45.797888 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41481
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:04:45.933409 14215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:45.933660 14215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:45.933733 14215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:45.937649 14215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:45.937736 14215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:45.937826 14215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:04:45.942919 14215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:04:45.943950 14215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:45.944913 14215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:45.954350 14221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:45.954414 14220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:45.954543 14215 server_base.cc:1034] running on GCE node
W20260929 01:04:45.954639 14223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:45.955101 14215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:45.955811 14215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:45.956969 14215 hybrid_clock.cc:648] HybridClock initialized: now 1790643885956953 us; error 25 us; skew 500 ppm
Sep 29 01:04:45 dist-test-slave-7zdl krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:45.959705 14215 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:04:45.960479 14215 webserver.cc:460] Webserver started at http://127.0.98.1:35165/ using document root <none> and password file <none>
I20260929 01:04:45.960969 14215 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:45.961025 14215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:45.961203 14215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:04:45.963160 14215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "aee98792440b4cad86a9757cd709a314"
format_stamp: "Formatted at 2026-09-29 01:04:45 on dist-test-slave-7zdl"
server_key: "be75733d1e65f0811a754e23dbfb589c"
server_key_iv: "e0696b1333a782b9be0e58eb3f3804ba"
server_key_version: "encryptionkey@0"
I20260929 01:04:45.963644 14215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "aee98792440b4cad86a9757cd709a314"
format_stamp: "Formatted at 2026-09-29 01:04:45 on dist-test-slave-7zdl"
server_key: "be75733d1e65f0811a754e23dbfb589c"
server_key_iv: "e0696b1333a782b9be0e58eb3f3804ba"
server_key_version: "encryptionkey@0"
I20260929 01:04:45.967623 14215 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:04:45.969789 14230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:45.971025 14215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:04:45.971149 14215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "aee98792440b4cad86a9757cd709a314"
format_stamp: "Formatted at 2026-09-29 01:04:45 on dist-test-slave-7zdl"
server_key: "be75733d1e65f0811a754e23dbfb589c"
server_key_iv: "e0696b1333a782b9be0e58eb3f3804ba"
server_key_version: "encryptionkey@0"
I20260929 01:04:45.971244 14215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:45.990749 14215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:45.993983 14215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:45.994199 14215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:45.994848 14215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:45.995648 14215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:45.995718 14215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:45.995782 14215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:45.995833 14215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:46.005191 14215 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33439
I20260929 01:04:46.005208 14343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33439 every 8 connection(s)
I20260929 01:04:46.006136 14215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:04:46.014781 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 14215
I20260929 01:04:46.014955 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
Sep 29 01:04:46 dist-test-slave-7zdl krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:04:46.015272 392 external_mini_cluster.cc:1428] Setting key 945f5917344fdaab305f6409f1d172b6
I20260929 01:04:46.020068 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:46.008009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42043 (local address 127.0.98.62:41481)
0929 01:04:46.008147 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:46.008151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:46.009146 (+ 995us) server_negotiation.cc:407] Connection header received
0929 01:04:46.009940 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:46.009943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:46.009992 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:46.010077 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:46.011241 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.011790 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.012922 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.013153 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.013238 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:46.016018 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:46.016021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:46.016024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:46.016047 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:46.017832 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:46.018320 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:46.018326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:46.018330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:46.018429 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:46.018879 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:46.018881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:46.018882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:46.019047 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:46.019126 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:46.019782 (+ 656us) server_negotiation.cc:299] Negotiation successful
0929 01:04:46.019921 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 01:04:46.021159 14346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:46.008384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.1:42043)
0929 01:04:46.009001 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:46.009040 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:46.009756 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:46.010231 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:46.010237 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:46.010583 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:46.011108 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:46.011120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.011932 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.011935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:46.012810 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:46.012819 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.013298 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.013300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:46.013434 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:46.013936 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:46.013950 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:46.015851 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:46.017957 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:46.017962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:46.017974 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:46.018219 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:46.018541 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:46.018546 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:46.018550 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:46.018740 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:46.019151 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:46.019161 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:46.019485 (+ 324us) client_negotiation.cc:769] Sending connection context
0929 01:04:46.019696 (+ 211us) client_negotiation.cc:240] Negotiation successful
0929 01:04:46.020064 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":122,"threads_started":1}
I20260929 01:04:46.022456 14344 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41481
I20260929 01:04:46.022743 14344 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:46.023393 14344 heartbeater.cc:511] Master 127.0.98.62:41481 requested a full tablet report, sending...
I20260929 01:04:46.025085 14153 ts_manager.cc:194] Registered new tserver with Master: aee98792440b4cad86a9757cd709a314 (127.0.98.1:33439)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 01:04:46.026554 14153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42043
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 01:04:46.043133 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41481
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:04:46.152643 14349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:46.152870 14349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:46.152911 14349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:46.156962 14349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:46.157014 14349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:46.157073 14349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:04:46.162205 14349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:04:46.163178 14349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:46.164093 14349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:46.173564 14354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:46.173595 14355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:46.173775 14357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:46.174089 14349 server_base.cc:1034] running on GCE node
I20260929 01:04:46.174464 14349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:46.175114 14349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:46.176299 14349 hybrid_clock.cc:648] HybridClock initialized: now 1790643886176285 us; error 32 us; skew 500 ppm
Sep 29 01:04:46 dist-test-slave-7zdl krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:46.179068 14349 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:04:46.179847 14349 webserver.cc:460] Webserver started at http://127.0.98.2:46085/ using document root <none> and password file <none>
I20260929 01:04:46.180353 14349 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:46.180447 14349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:46.180625 14349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:04:46.182548 14349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "80ebf772f86d4df396ff05b6b433e785"
format_stamp: "Formatted at 2026-09-29 01:04:46 on dist-test-slave-7zdl"
server_key: "cd3687598f92a752ab0d32b49adac660"
server_key_iv: "421e3f2eae0d3b2a5776698d71a7b2c0"
server_key_version: "encryptionkey@0"
I20260929 01:04:46.183089 14349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "80ebf772f86d4df396ff05b6b433e785"
format_stamp: "Formatted at 2026-09-29 01:04:46 on dist-test-slave-7zdl"
server_key: "cd3687598f92a752ab0d32b49adac660"
server_key_iv: "421e3f2eae0d3b2a5776698d71a7b2c0"
server_key_version: "encryptionkey@0"
I20260929 01:04:46.186627 14349 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:04:46.188915 14364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:46.190156 14349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:04:46.190265 14349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "80ebf772f86d4df396ff05b6b433e785"
format_stamp: "Formatted at 2026-09-29 01:04:46 on dist-test-slave-7zdl"
server_key: "cd3687598f92a752ab0d32b49adac660"
server_key_iv: "421e3f2eae0d3b2a5776698d71a7b2c0"
server_key_version: "encryptionkey@0"
I20260929 01:04:46.190341 14349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:46.202865 14349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:46.205418 14349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:46.205560 14349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:46.206033 14349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:46.206753 14349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:46.206845 14349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:46.206907 14349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:46.206930 14349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:46.215677 14349 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37059
I20260929 01:04:46.215703 14477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37059 every 8 connection(s)
I20260929 01:04:46.216531 14349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:04:46.218603 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 14349
I20260929 01:04:46.218743 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:04:46.219089 392 external_mini_cluster.cc:1428] Setting key e71cad73a5b88d788127189eb0f0ec4a
Sep 29 01:04:46 dist-test-slave-7zdl krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 01:04:46.230234 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:46.218250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51415 (local address 127.0.98.62:41481)
0929 01:04:46.218402 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:46.218406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:46.219593 (+ 1187us) server_negotiation.cc:407] Connection header received
0929 01:04:46.220455 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:46.220457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:46.220494 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:46.220587 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:46.221675 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.222401 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.223847 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.224155 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.224277 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:46.226707 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:46.226711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:46.226721 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:46.226744 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:46.228218 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:46.228720 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:46.228723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:46.228725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:46.228806 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:46.229133 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:46.229137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:46.229139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:46.229314 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:46.229443 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:46.230008 (+ 565us) server_negotiation.cc:299] Negotiation successful
0929 01:04:46.230105 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 01:04:46.231312 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:46.218653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.2:51415)
0929 01:04:46.219407 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:46.219474 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:46.220250 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:46.220753 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:46.220759 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:46.221101 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:46.221533 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:46.221545 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.222526 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.222533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:46.223714 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:46.223724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.224313 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.224318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:46.224495 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:46.225100 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:46.225116 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:46.226552 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:46.228334 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:46.228340 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:46.228355 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:46.228601 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:46.228896 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:46.228900 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:46.228902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:46.229034 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:46.229462 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:46.229468 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:46.229781 (+ 313us) client_negotiation.cc:769] Sending connection context
0929 01:04:46.229976 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 01:04:46.230267 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":136,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 01:04:46.232555 14478 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41481
I20260929 01:04:46.232820 14478 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:46.233294 14478 heartbeater.cc:511] Master 127.0.98.62:41481 requested a full tablet report, sending...
I20260929 01:04:46.234290 14153 ts_manager.cc:194] Registered new tserver with Master: 80ebf772f86d4df396ff05b6b433e785 (127.0.98.2:37059)
I20260929 01:04:46.234915 14153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51415
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 01:04:46.247365 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41481
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:04:46.355518 14483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:46.355811 14483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:46.355882 14483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:46.360004 14483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:46.360054 14483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:46.360134 14483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:04:46.365298 14483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:04:46.366293 14483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:46.367341 14483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:46.376394 14491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:46.376402 14488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:46.376509 14483 server_base.cc:1034] running on GCE node
W20260929 01:04:46.376410 14489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:46.377071 14483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:46.377709 14483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:46.378871 14483 hybrid_clock.cc:648] HybridClock initialized: now 1790643886378844 us; error 48 us; skew 500 ppm
Sep 29 01:04:46 dist-test-slave-7zdl krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:46.381474 14483 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:04:46.382273 14483 webserver.cc:460] Webserver started at http://127.0.98.3:40517/ using document root <none> and password file <none>
I20260929 01:04:46.382907 14483 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:46.382972 14483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:46.383189 14483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:04:46.385237 14483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "50409a27acc24ff0b455aa410bfd076d"
format_stamp: "Formatted at 2026-09-29 01:04:46 on dist-test-slave-7zdl"
server_key: "3c7422f16c4a053f331f0634b3cdda50"
server_key_iv: "34f4b76f7c6b5446ad98a32ec391517b"
server_key_version: "encryptionkey@0"
I20260929 01:04:46.385783 14483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "50409a27acc24ff0b455aa410bfd076d"
format_stamp: "Formatted at 2026-09-29 01:04:46 on dist-test-slave-7zdl"
server_key: "3c7422f16c4a053f331f0634b3cdda50"
server_key_iv: "34f4b76f7c6b5446ad98a32ec391517b"
server_key_version: "encryptionkey@0"
I20260929 01:04:46.389328 14483 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 01:04:46.391618 14498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:46.392802 14483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260929 01:04:46.392910 14483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "50409a27acc24ff0b455aa410bfd076d"
format_stamp: "Formatted at 2026-09-29 01:04:46 on dist-test-slave-7zdl"
server_key: "3c7422f16c4a053f331f0634b3cdda50"
server_key_iv: "34f4b76f7c6b5446ad98a32ec391517b"
server_key_version: "encryptionkey@0"
I20260929 01:04:46.393005 14483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:46.413149 14483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:46.416018 14483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:46.416204 14483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:46.416870 14483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:46.417681 14483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:46.417727 14483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:46.417801 14483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:46.417840 14483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:46.426849 14483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33801
I20260929 01:04:46.426914 14611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33801 every 8 connection(s)
I20260929 01:04:46.427719 14483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:04:46.433202 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 14483
I20260929 01:04:46.433303 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:04:46.433578 392 external_mini_cluster.cc:1428] Setting key 165e08db46602f1519352c1e99e7f07a
Sep 29 01:04:46 dist-test-slave-7zdl krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:04:46.441424 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:46.429474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55781 (local address 127.0.98.62:41481)
0929 01:04:46.429611 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:46.429614 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:46.431233 (+ 1619us) server_negotiation.cc:407] Connection header received
0929 01:04:46.432447 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:46.432451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:46.432530 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:46.432645 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:46.433969 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.434736 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.436008 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.436220 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.436319 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:46.438522 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:46.438525 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:46.438528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:46.438551 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:46.439788 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:46.440214 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:46.440217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:46.440218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:46.440300 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:46.440569 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:46.440571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:46.440572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:46.440725 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:46.440796 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:46.441204 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 01:04:46.441308 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 01:04:46.442176 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:46.429859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.3:55781)
0929 01:04:46.430958 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:46.431081 (+ 123us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:46.432225 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:46.432827 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:46.432834 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:46.433200 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:46.433801 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:46.433817 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.434906 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.434909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:46.435906 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:46.435915 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.436358 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.436361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:46.436491 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:46.436993 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:46.437008 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:46.438383 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:46.439891 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:46.439895 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:46.439906 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:46.440133 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:46.440376 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:46.440378 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:46.440379 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:46.440496 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:46.440805 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:46.440809 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:46.441019 (+ 210us) client_negotiation.cc:769] Sending connection context
0929 01:04:46.441161 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 01:04:46.441396 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"thread_start_us":609,"threads_started":1}
I20260929 01:04:46.443218 14612 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41481
I20260929 01:04:46.443449 14612 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:46.443884 14612 heartbeater.cc:511] Master 127.0.98.62:41481 requested a full tablet report, sending...
I20260929 01:04:46.444839 14153 ts_manager.cc:194] Registered new tserver with Master: 50409a27acc24ff0b455aa410bfd076d (127.0.98.3:33801)
I20260929 01:04:46.445412 14153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55781
I20260929 01:04:46.447329 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:04:46.455896 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:46.449206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43766 (local address 127.0.98.62:41481)
0929 01:04:46.449317 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:46.449320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:46.449495 (+ 175us) server_negotiation.cc:407] Connection header received
0929 01:04:46.449596 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:46.449598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:46.449630 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:46.449696 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:46.450290 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.450767 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.451594 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.451794 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.451871 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:46.453036 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:46.453039 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:46.453040 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:46.453060 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:46.454470 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:46.454868 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:46.454871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:46.454872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:46.454939 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:46.455158 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:46.455160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:46.455161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:46.455401 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:46.455503 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:46.455672 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 01:04:46.455757 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 01:04:46.458438 14153 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:04:46.461465 14153 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:04:46.475656 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:46.469525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35546 (local address 127.0.98.1:33439)
0929 01:04:46.470412 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:46.470424 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:46.470467 (+ 43us) server_negotiation.cc:407] Connection header received
0929 01:04:46.470548 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:46.470554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:46.470777 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:46.471021 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:46.471326 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.472509 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.474229 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.474992 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.475098 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:46.475248 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:46.475426 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 01:04:46.475512 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":138,"threads_started":1}
I20260929 01:04:46.475644 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:46.469349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33439 (local address 127.0.0.1:35546)
0929 01:04:46.470113 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:46.470128 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:46.470237 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:46.471025 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:46.471031 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:46.471054 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:46.471211 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:46.471218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.472643 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.472646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:46.474093 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:46.474103 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.475107 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.475110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:46.475219 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:46.475255 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 01:04:46.475391 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 01:04:46.475505 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":61,"thread_start_us":62,"threads_started":1}
I20260929 01:04:46.475857 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:46.469755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37059 (local address 127.0.0.1:60624)
0929 01:04:46.470048 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:46.470109 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:46.470237 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:46.471381 (+ 1144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:46.471384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:46.471401 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:46.471512 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:46.471520 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.472901 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.472906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:46.474058 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:46.474068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.475375 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.475381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:46.475483 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:46.475505 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 01:04:46.475595 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 01:04:46.475664 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":2,"thread_start_us":97,"threads_started":1}
I20260929 01:04:46.476015 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:46.470010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33801 (local address 127.0.0.1:38300)
0929 01:04:46.470734 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:46.470745 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:46.471550 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:46.471953 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:46.471956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:46.471971 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:46.472071 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:46.472078 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.473219 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.473222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:46.474338 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:46.474345 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.475611 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.475615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:46.475705 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:46.475727 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 01:04:46.475814 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 01:04:46.475885 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":37,"thread_start_us":70,"threads_started":1}
I20260929 01:04:46.476442 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:46.470375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38300 (local address 127.0.98.3:33801)
0929 01:04:46.470644 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:46.470657 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:46.470874 (+ 217us) server_negotiation.cc:407] Connection header received
0929 01:04:46.471707 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:46.471711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:46.471834 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:46.471972 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:46.472188 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.473074 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.474451 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.475318 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.475423 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:46.475551 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:46.476183 (+ 632us) server_negotiation.cc:299] Negotiation successful
0929 01:04:46.476279 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":104,"threads_started":1}
I20260929 01:04:46.476904 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:46.469972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60624 (local address 127.0.98.2:37059)
0929 01:04:46.471018 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:46.471030 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:46.471051 (+ 21us) server_negotiation.cc:407] Connection header received
0929 01:04:46.471122 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:46.471125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:46.471249 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:46.471393 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:46.471644 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.472773 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.474183 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.475053 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.475168 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:46.475304 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:46.476030 (+ 726us) server_negotiation.cc:299] Negotiation successful
0929 01:04:46.476146 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":949,"thread_start_us":120,"threads_started":1}
I20260929 01:04:46.479269 14412 tablet_service.cc:1467] Processing CreateTablet for tablet e9b40ccc554e4c3795c48a0e44371896 (DEFAULT_TABLE table=test-table [id=dfe2d1e0e81848ba92bcd8dd7cd7f248]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:04:46.479254 14546 tablet_service.cc:1467] Processing CreateTablet for tablet e9b40ccc554e4c3795c48a0e44371896 (DEFAULT_TABLE table=test-table [id=dfe2d1e0e81848ba92bcd8dd7cd7f248]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:04:46.479250 14278 tablet_service.cc:1467] Processing CreateTablet for tablet e9b40ccc554e4c3795c48a0e44371896 (DEFAULT_TABLE table=test-table [id=dfe2d1e0e81848ba92bcd8dd7cd7f248]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:04:46.480283 14278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9b40ccc554e4c3795c48a0e44371896. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:46.480286 14546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9b40ccc554e4c3795c48a0e44371896. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:46.480293 14412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9b40ccc554e4c3795c48a0e44371896. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:46.486567 14629 tablet_bootstrap.cc:492] T e9b40ccc554e4c3795c48a0e44371896 P 80ebf772f86d4df396ff05b6b433e785: Bootstrap starting.
I20260929 01:04:46.487881 14631 tablet_bootstrap.cc:492] T e9b40ccc554e4c3795c48a0e44371896 P 50409a27acc24ff0b455aa410bfd076d: Bootstrap starting.
I20260929 01:04:46.488009 14630 tablet_bootstrap.cc:492] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314: Bootstrap starting.
I20260929 01:04:46.489156 14629 tablet_bootstrap.cc:654] T e9b40ccc554e4c3795c48a0e44371896 P 80ebf772f86d4df396ff05b6b433e785: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:46.490005 14630 tablet_bootstrap.cc:654] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:46.490005 14631 tablet_bootstrap.cc:654] T e9b40ccc554e4c3795c48a0e44371896 P 50409a27acc24ff0b455aa410bfd076d: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:46.490231 14629 log.cc:824] T e9b40ccc554e4c3795c48a0e44371896 P 80ebf772f86d4df396ff05b6b433e785: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:46.491328 14631 log.cc:824] T e9b40ccc554e4c3795c48a0e44371896 P 50409a27acc24ff0b455aa410bfd076d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:46.491328 14630 log.cc:824] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:46.492781 14629 tablet_bootstrap.cc:492] T e9b40ccc554e4c3795c48a0e44371896 P 80ebf772f86d4df396ff05b6b433e785: No bootstrap required, opened a new log
I20260929 01:04:46.492992 14629 ts_tablet_manager.cc:1397] T e9b40ccc554e4c3795c48a0e44371896 P 80ebf772f86d4df396ff05b6b433e785: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 01:04:46.493137 14630 tablet_bootstrap.cc:492] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314: No bootstrap required, opened a new log
I20260929 01:04:46.493304 14630 ts_tablet_manager.cc:1397] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 01:04:46.493530 14631 tablet_bootstrap.cc:492] T e9b40ccc554e4c3795c48a0e44371896 P 50409a27acc24ff0b455aa410bfd076d: No bootstrap required, opened a new log
I20260929 01:04:46.493701 14631 ts_tablet_manager.cc:1397] T e9b40ccc554e4c3795c48a0e44371896 P 50409a27acc24ff0b455aa410bfd076d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 01:04:46.496244 14629 raft_consensus.cc:348] T e9b40ccc554e4c3795c48a0e44371896 P 80ebf772f86d4df396ff05b6b433e785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } }
I20260929 01:04:46.496654 14629 raft_consensus.cc:374] T e9b40ccc554e4c3795c48a0e44371896 P 80ebf772f86d4df396ff05b6b433e785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:46.496721 14629 raft_consensus.cc:729] T e9b40ccc554e4c3795c48a0e44371896 P 80ebf772f86d4df396ff05b6b433e785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80ebf772f86d4df396ff05b6b433e785, State: Initialized, Role: FOLLOWER
I20260929 01:04:46.497077 14630 raft_consensus.cc:348] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } }
I20260929 01:04:46.497186 14629 consensus_queue.cc:260] T e9b40ccc554e4c3795c48a0e44371896 P 80ebf772f86d4df396ff05b6b433e785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } }
I20260929 01:04:46.497438 14630 raft_consensus.cc:374] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:46.497532 14630 raft_consensus.cc:729] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee98792440b4cad86a9757cd709a314, State: Initialized, Role: FOLLOWER
I20260929 01:04:46.497411 14631 raft_consensus.cc:348] T e9b40ccc554e4c3795c48a0e44371896 P 50409a27acc24ff0b455aa410bfd076d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } }
I20260929 01:04:46.497720 14631 raft_consensus.cc:374] T e9b40ccc554e4c3795c48a0e44371896 P 50409a27acc24ff0b455aa410bfd076d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:46.497807 14631 raft_consensus.cc:729] T e9b40ccc554e4c3795c48a0e44371896 P 50409a27acc24ff0b455aa410bfd076d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50409a27acc24ff0b455aa410bfd076d, State: Initialized, Role: FOLLOWER
I20260929 01:04:46.498029 14630 consensus_queue.cc:260] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } }
I20260929 01:04:46.498253 14631 consensus_queue.cc:260] T e9b40ccc554e4c3795c48a0e44371896 P 50409a27acc24ff0b455aa410bfd076d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } }
I20260929 01:04:46.498312 14629 ts_tablet_manager.cc:1428] T e9b40ccc554e4c3795c48a0e44371896 P 80ebf772f86d4df396ff05b6b433e785: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 01:04:46.498688 14478 heartbeater.cc:503] Master 127.0.98.62:41481 was elected leader, sending a full tablet report...
I20260929 01:04:46.498695 14344 heartbeater.cc:503] Master 127.0.98.62:41481 was elected leader, sending a full tablet report...
I20260929 01:04:46.499118 14631 ts_tablet_manager.cc:1428] T e9b40ccc554e4c3795c48a0e44371896 P 50409a27acc24ff0b455aa410bfd076d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 01:04:46.499416 14612 heartbeater.cc:503] Master 127.0.98.62:41481 was elected leader, sending a full tablet report...
I20260929 01:04:46.500290 14630 ts_tablet_manager.cc:1428] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260929 01:04:46.507278 14345 tablet.cc:2369] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:04:46.678697 14613 tablet.cc:2369] T e9b40ccc554e4c3795c48a0e44371896 P 50409a27acc24ff0b455aa410bfd076d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:04:46.717605 14479 tablet.cc:2369] T e9b40ccc554e4c3795c48a0e44371896 P 80ebf772f86d4df396ff05b6b433e785: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:04:46.753826 14636 raft_consensus.cc:482] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:04:46.754022 14636 raft_consensus.cc:504] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } }
I20260929 01:04:46.755129 14636 leader_election.cc:288] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50409a27acc24ff0b455aa410bfd076d (127.0.98.3:33801), 80ebf772f86d4df396ff05b6b433e785 (127.0.98.2:37059)
I20260929 01:04:46.759488 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:46.755537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49939 (local address 127.0.98.2:37059)
0929 01:04:46.755684 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:46.755689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:46.755890 (+ 201us) server_negotiation.cc:407] Connection header received
0929 01:04:46.756037 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:46.756039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:46.756083 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:46.756177 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:46.756578 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.757365 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.758499 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.758928 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.759047 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:46.759115 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:46.759284 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 01:04:46.759358 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 01:04:46.759506 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:46.755388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37059 (local address 127.0.98.1:49939)
0929 01:04:46.755784 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:46.755798 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:46.755906 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:46.756210 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:46.756213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:46.756229 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:46.756370 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:46.756377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.757476 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.757479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:46.758381 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:46.758390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.759047 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.759051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:46.759150 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:46.759184 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 01:04:46.759269 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 01:04:46.759335 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":36,"thread_start_us":96,"threads_started":1}
I20260929 01:04:46.759533 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:46.755544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59207 (local address 127.0.98.3:33801)
0929 01:04:46.755788 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:46.755794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:46.755817 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:04:46.756010 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:46.756013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:46.756066 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:46.756123 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:46.756567 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.757361 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.758481 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.758988 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.759107 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:46.759169 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:46.759346 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 01:04:46.759419 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260929 01:04:46.759552 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:46.755388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33801 (local address 127.0.98.1:59207)
0929 01:04:46.755732 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:46.755755 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:46.755878 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:46.756163 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:46.756168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:46.756184 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:46.756433 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:46.756445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.757516 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.757519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:46.758342 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:46.758350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.759107 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.759111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:46.759210 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:46.759233 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 01:04:46.759330 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 01:04:46.759397 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":29,"thread_start_us":121,"threads_started":1}
I20260929 01:04:46.760160 14432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b40ccc554e4c3795c48a0e44371896" candidate_uuid: "aee98792440b4cad86a9757cd709a314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80ebf772f86d4df396ff05b6b433e785" is_pre_election: true
I20260929 01:04:46.760305 14566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b40ccc554e4c3795c48a0e44371896" candidate_uuid: "aee98792440b4cad86a9757cd709a314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50409a27acc24ff0b455aa410bfd076d" is_pre_election: true
I20260929 01:04:46.760484 14432 raft_consensus.cc:2454] T e9b40ccc554e4c3795c48a0e44371896 P 80ebf772f86d4df396ff05b6b433e785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aee98792440b4cad86a9757cd709a314 in term 0.
I20260929 01:04:46.760550 14566 raft_consensus.cc:2454] T e9b40ccc554e4c3795c48a0e44371896 P 50409a27acc24ff0b455aa410bfd076d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aee98792440b4cad86a9757cd709a314 in term 0.
I20260929 01:04:46.760978 14231 leader_election.cc:302] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50409a27acc24ff0b455aa410bfd076d, aee98792440b4cad86a9757cd709a314; no voters:
I20260929 01:04:46.761264 14636 raft_consensus.cc:2781] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:04:46.761359 14636 raft_consensus.cc:482] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:04:46.761418 14636 raft_consensus.cc:3037] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:46.762441 14636 raft_consensus.cc:504] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } }
I20260929 01:04:46.762778 14636 leader_election.cc:288] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314 [CANDIDATE]: Term 1 election: Requested vote from peers 50409a27acc24ff0b455aa410bfd076d (127.0.98.3:33801), 80ebf772f86d4df396ff05b6b433e785 (127.0.98.2:37059)
I20260929 01:04:46.763157 14566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b40ccc554e4c3795c48a0e44371896" candidate_uuid: "aee98792440b4cad86a9757cd709a314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50409a27acc24ff0b455aa410bfd076d"
I20260929 01:04:46.763167 14432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b40ccc554e4c3795c48a0e44371896" candidate_uuid: "aee98792440b4cad86a9757cd709a314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80ebf772f86d4df396ff05b6b433e785"
I20260929 01:04:46.763293 14566 raft_consensus.cc:3037] T e9b40ccc554e4c3795c48a0e44371896 P 50409a27acc24ff0b455aa410bfd076d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:46.763309 14432 raft_consensus.cc:3037] T e9b40ccc554e4c3795c48a0e44371896 P 80ebf772f86d4df396ff05b6b433e785 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:46.764518 14566 raft_consensus.cc:2454] T e9b40ccc554e4c3795c48a0e44371896 P 50409a27acc24ff0b455aa410bfd076d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aee98792440b4cad86a9757cd709a314 in term 1.
I20260929 01:04:46.764518 14432 raft_consensus.cc:2454] T e9b40ccc554e4c3795c48a0e44371896 P 80ebf772f86d4df396ff05b6b433e785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aee98792440b4cad86a9757cd709a314 in term 1.
I20260929 01:04:46.764773 14231 leader_election.cc:302] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50409a27acc24ff0b455aa410bfd076d, aee98792440b4cad86a9757cd709a314; no voters:
I20260929 01:04:46.764940 14636 raft_consensus.cc:2781] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:04:46.765165 14636 raft_consensus.cc:686] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314 [term 1 LEADER]: Becoming Leader. State: Replica: aee98792440b4cad86a9757cd709a314, State: Running, Role: LEADER
I20260929 01:04:46.765473 14636 consensus_queue.cc:237] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } }
I20260929 01:04:46.768378 14151 catalog_manager.cc:5495] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314 reported cstate change: term changed from 0 to 1, leader changed from <none> to aee98792440b4cad86a9757cd709a314 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "aee98792440b4cad86a9757cd709a314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } health_report { overall_health: UNKNOWN } } }
I20260929 01:04:46.835767 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:46.832447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35562 (local address 127.0.98.1:33439)
0929 01:04:46.832578 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:46.832590 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:46.832603 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:04:46.832684 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:46.832687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:46.832741 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:46.832829 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:04:46.833048 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.833558 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.834404 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:46.834605 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:46.834681 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:46.834964 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:04:46.835387 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:04:46.835480 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:46.835541 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 01:04:46.835591 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 01:04:46.841485 14432 raft_consensus.cc:1261] T e9b40ccc554e4c3795c48a0e44371896 P 80ebf772f86d4df396ff05b6b433e785 [term 1 FOLLOWER]: Refusing update from remote peer aee98792440b4cad86a9757cd709a314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:04:46.841485 14566 raft_consensus.cc:1261] T e9b40ccc554e4c3795c48a0e44371896 P 50409a27acc24ff0b455aa410bfd076d [term 1 FOLLOWER]: Refusing update from remote peer aee98792440b4cad86a9757cd709a314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:04:46.842218 14640 consensus_queue.cc:1035] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:04:46.842417 14636 consensus_queue.cc:1035] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:04:46.857440 14153 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:04:46.857627 14153 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:04:46.860080 14153 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e: Sending DeleteTablet for 3 replicas of tablet e9b40ccc554e4c3795c48a0e44371896
I20260929 01:04:46.860903 14278 tablet_service.cc:1514] Processing DeleteTablet for tablet e9b40ccc554e4c3795c48a0e44371896 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:04:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35546
I20260929 01:04:46.860873 14412 tablet_service.cc:1514] Processing DeleteTablet for tablet e9b40ccc554e4c3795c48a0e44371896 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:04:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60624
I20260929 01:04:46.861364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 14215
I20260929 01:04:46.861370 14652 tablet_replica.cc:330] T e9b40ccc554e4c3795c48a0e44371896 P 80ebf772f86d4df396ff05b6b433e785: stopping tablet replica
I20260929 01:04:46.861662 14652 raft_consensus.cc:2229] T e9b40ccc554e4c3795c48a0e44371896 P 80ebf772f86d4df396ff05b6b433e785 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:04:46.861991 14652 raft_consensus.cc:2258] T e9b40ccc554e4c3795c48a0e44371896 P 80ebf772f86d4df396ff05b6b433e785 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:04:46.863336 14652 ts_tablet_manager.cc:1905] T e9b40ccc554e4c3795c48a0e44371896 P 80ebf772f86d4df396ff05b6b433e785: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:04:46.864204 14546 tablet_service.cc:1514] Processing DeleteTablet for tablet e9b40ccc554e4c3795c48a0e44371896 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:04:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38300
I20260929 01:04:46.864687 14654 tablet_replica.cc:330] T e9b40ccc554e4c3795c48a0e44371896 P 50409a27acc24ff0b455aa410bfd076d: stopping tablet replica
I20260929 01:04:46.865010 14654 raft_consensus.cc:2229] T e9b40ccc554e4c3795c48a0e44371896 P 50409a27acc24ff0b455aa410bfd076d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:04:46.865294 14654 raft_consensus.cc:2258] T e9b40ccc554e4c3795c48a0e44371896 P 50409a27acc24ff0b455aa410bfd076d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:04:46.866389 14652 ts_tablet_manager.cc:1918] T e9b40ccc554e4c3795c48a0e44371896 P 80ebf772f86d4df396ff05b6b433e785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:04:46.866489 14652 log.cc:1196] T e9b40ccc554e4c3795c48a0e44371896 P 80ebf772f86d4df396ff05b6b433e785: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/wal/wals/e9b40ccc554e4c3795c48a0e44371896
I20260929 01:04:46.866622 14654 ts_tablet_manager.cc:1905] T e9b40ccc554e4c3795c48a0e44371896 P 50409a27acc24ff0b455aa410bfd076d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:04:46.866885 14652 ts_tablet_manager.cc:1939] T e9b40ccc554e4c3795c48a0e44371896 P 80ebf772f86d4df396ff05b6b433e785: Deleting consensus metadata
I20260929 01:04:46.868155 14138 catalog_manager.cc:4841] TS 80ebf772f86d4df396ff05b6b433e785 (127.0.98.2:37059): tablet e9b40ccc554e4c3795c48a0e44371896 (table test-table [id=dfe2d1e0e81848ba92bcd8dd7cd7f248]) successfully deleted
W20260929 01:04:46.869331 14137 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33439 (error 108)
I20260929 01:04:46.869381 14654 ts_tablet_manager.cc:1918] T e9b40ccc554e4c3795c48a0e44371896 P 50409a27acc24ff0b455aa410bfd076d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:04:46.869460 14654 log.cc:1196] T e9b40ccc554e4c3795c48a0e44371896 P 50409a27acc24ff0b455aa410bfd076d: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/e9b40ccc554e4c3795c48a0e44371896
I20260929 01:04:46.869475 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 14349
I20260929 01:04:46.870857 14654 ts_tablet_manager.cc:1939] T e9b40ccc554e4c3795c48a0e44371896 P 50409a27acc24ff0b455aa410bfd076d: Deleting consensus metadata
I20260929 01:04:46.871663 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:46.871329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33439 (local address 127.0.0.1:35574)
0929 01:04:46.871480 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:46.871560 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 01:04:46.871894 14137 catalog_manager.cc:4584] TS aee98792440b4cad86a9757cd709a314 (127.0.98.1:33439): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e9b40ccc554e4c3795c48a0e44371896: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33439: connect: Connection refused (error 111)
I20260929 01:04:46.872115 14137 catalog_manager.cc:4841] TS 50409a27acc24ff0b455aa410bfd076d (127.0.98.3:33801): tablet e9b40ccc554e4c3795c48a0e44371896 (table test-table [id=dfe2d1e0e81848ba92bcd8dd7cd7f248]) successfully deleted
I20260929 01:04:46.875336 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 14483
I20260929 01:04:46.880838 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 14121
I20260929 01:04:46.887172 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41481
--webserver_interface=127.0.98.62
--webserver_port=41195
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:04:46.995388 14656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:46.995649 14656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:46.995719 14656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:46.999698 14656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:04:46.999761 14656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:46.999781 14656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:46.999838 14656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:04:46.999867 14656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:04:47.005162 14656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:04:47.006213 14656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:47.007244 14656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:47.020009 14665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:47.020224 14656 server_base.cc:1034] running on GCE node
W20260929 01:04:47.020009 14662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:47.020431 14663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:47.021090 14656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:47.022243 14656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:47.023422 14656 hybrid_clock.cc:648] HybridClock initialized: now 1790643887023395 us; error 60 us; skew 500 ppm
Sep 29 01:04:47 dist-test-slave-7zdl krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:47.026072 14656 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:04:47.026970 14656 webserver.cc:460] Webserver started at http://127.0.98.62:41195/ using document root <none> and password file <none>
I20260929 01:04:47.027532 14656 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:47.027634 14656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:47.031332 14656 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 01:04:47.033385 14672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:47.034557 14656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:04:47.034701 14656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "72a783dac216456a9482b052f226ff5e"
format_stamp: "Formatted at 2026-09-29 01:04:45 on dist-test-slave-7zdl"
server_key: "13e1231a8fe278fc6d740d0415c80d3f"
server_key_iv: "8ce6e14e470b131a752cf61f97d0b3f4"
server_key_version: "encryptionkey@0"
I20260929 01:04:47.035346 14656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:47.074909 14656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:47.078709 14656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:47.078914 14656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:47.087212 14656 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41481
I20260929 01:04:47.087502 14724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41481 every 8 connection(s)
I20260929 01:04:47.088328 14656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:04:47.092624 14725 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:04:47.093796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 14656
I20260929 01:04:47.094277 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33439
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41481
--webserver_port=35165
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:04:47.094523 14725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e: Bootstrap starting.
I20260929 01:04:47.111482 14725 log.cc:824] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:47.119935 14725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:04:47.120321 14725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e: Bootstrap complete.
I20260929 01:04:47.124388 14725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a783dac216456a9482b052f226ff5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41481 } }
I20260929 01:04:47.124778 14725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72a783dac216456a9482b052f226ff5e, State: Initialized, Role: FOLLOWER
I20260929 01:04:47.125301 14725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a783dac216456a9482b052f226ff5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41481 } }
I20260929 01:04:47.125419 14725 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:04:47.125509 14725 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:04:47.125634 14725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:04:47.127542 14725 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a783dac216456a9482b052f226ff5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41481 } }
I20260929 01:04:47.127875 14725 leader_election.cc:302] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72a783dac216456a9482b052f226ff5e; no voters:
I20260929 01:04:47.128234 14725 leader_election.cc:288] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 01:04:47.128350 14729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:04:47.128557 14729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [term 2 LEADER]: Becoming Leader. State: Replica: 72a783dac216456a9482b052f226ff5e, State: Running, Role: LEADER
I20260929 01:04:47.128834 14729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a783dac216456a9482b052f226ff5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41481 } }
I20260929 01:04:47.129411 14725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:04:47.130075 14730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "72a783dac216456a9482b052f226ff5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a783dac216456a9482b052f226ff5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41481 } } }
I20260929 01:04:47.130206 14730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:47.130901 14731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72a783dac216456a9482b052f226ff5e. Latest consensus state: current_term: 2 leader_uuid: "72a783dac216456a9482b052f226ff5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a783dac216456a9482b052f226ff5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41481 } } }
I20260929 01:04:47.131052 14731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:47.131302 14736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:04:47.135790 14736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:04:47.136358 14736 catalog_manager.cc:1259] Loaded cluster ID: 8f4804fedb6848e99acedd943c9d443d
I20260929 01:04:47.136413 14736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:04:47.137997 14736 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 01:04:47.142577 14736 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 01:04:47.149295 14736 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:04:47.150041 14736 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e: Loaded TSK: 0
I20260929 01:04:47.151270 14736 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 01:04:47.211374 14727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:47.211632 14727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:47.211723 14727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:47.215730 14727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:47.215808 14727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:47.215889 14727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:04:47.220961 14727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35165
--tserver_master_addrs=127.0.98.62:41481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:04:47.221944 14727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:47.222967 14727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:47.233426 14755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:47.233480 14753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:47.233428 14752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:47.234166 14727 server_base.cc:1034] running on GCE node
I20260929 01:04:47.234587 14727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:47.235268 14727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:47.236442 14727 hybrid_clock.cc:648] HybridClock initialized: now 1790643887236412 us; error 47 us; skew 500 ppm
Sep 29 01:04:47 dist-test-slave-7zdl krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:47.239303 14727 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:04:47.240278 14727 webserver.cc:460] Webserver started at http://127.0.98.1:35165/ using document root <none> and password file <none>
I20260929 01:04:47.240819 14727 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:47.240890 14727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:47.244736 14727 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:04:47.246888 14762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:47.248188 14727 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:04:47.248310 14727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "aee98792440b4cad86a9757cd709a314"
format_stamp: "Formatted at 2026-09-29 01:04:45 on dist-test-slave-7zdl"
server_key: "be75733d1e65f0811a754e23dbfb589c"
server_key_iv: "e0696b1333a782b9be0e58eb3f3804ba"
server_key_version: "encryptionkey@0"
I20260929 01:04:47.248872 14727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:47.274462 14727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:47.277871 14727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:47.278062 14727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:47.278652 14727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:47.279726 14769 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 01:04:47.283690 14727 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 01:04:47.283762 14727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 01:04:47.283857 14727 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 01:04:47.285455 14727 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 01:04:47.285523 14727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 01:04:47.285786 14769 tablet_bootstrap.cc:492] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314: Bootstrap starting.
I20260929 01:04:47.296463 14727 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33439
I20260929 01:04:47.296527 14876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33439 every 8 connection(s)
I20260929 01:04:47.297386 14727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:04:47.300681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 14727
I20260929 01:04:47.301335 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37059
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41481
--webserver_port=46085
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:04:47.304942 14769 log.cc:824] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:04:47 dist-test-slave-7zdl krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:04:47.314338 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:47.299505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38339 (local address 127.0.98.62:41481)
0929 01:04:47.300046 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:47.300063 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:47.300113 (+ 50us) server_negotiation.cc:407] Connection header received
0929 01:04:47.301275 (+ 1162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:47.301285 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:47.301683 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:47.302025 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:47.303400 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.304873 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.306309 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.306919 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.307096 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:47.309459 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:47.309463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:47.309474 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:47.309499 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:47.311238 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:47.311761 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:47.311766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:47.311772 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:47.311885 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:47.312222 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:47.312227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:47.312229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:47.312549 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:47.312688 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:47.313221 (+ 533us) server_negotiation.cc:299] Negotiation successful
0929 01:04:47.313467 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":145,"threads_started":1}
I20260929 01:04:47.314348 14879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:47.299457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.1:38339)
0929 01:04:47.299939 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:47.299992 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:47.300852 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:47.302086 (+ 1234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:47.302094 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:47.302509 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:47.303190 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:47.303212 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.305032 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.305056 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:47.306182 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:47.306192 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.307059 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.307064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:47.307258 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:47.307860 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:47.307876 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:47.309272 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:47.311372 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:47.311377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:47.311390 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:47.311638 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:47.311981 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:47.311985 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:47.311987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:47.312125 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:47.312681 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:47.312686 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:47.312925 (+ 239us) client_negotiation.cc:769] Sending connection context
0929 01:04:47.313104 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 01:04:47.313419 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":142,"threads_started":1}
I20260929 01:04:47.314640 14769 tablet_bootstrap.cc:492] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:04:47.315083 14769 tablet_bootstrap.cc:492] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314: Bootstrap complete.
I20260929 01:04:47.316490 14769 ts_tablet_manager.cc:1397] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314: Time spent bootstrapping tablet: real 0.031s user 0.025s sys 0.004s
I20260929 01:04:47.317206 14877 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41481
I20260929 01:04:47.317430 14877 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:47.317900 14877 heartbeater.cc:511] Master 127.0.98.62:41481 requested a full tablet report, sending...
I20260929 01:04:47.319407 14769 raft_consensus.cc:348] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } }
I20260929 01:04:47.319909 14769 raft_consensus.cc:729] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee98792440b4cad86a9757cd709a314, State: Initialized, Role: FOLLOWER
I20260929 01:04:47.320267 14682 ts_manager.cc:194] Registered new tserver with Master: aee98792440b4cad86a9757cd709a314 (127.0.98.1:33439)
I20260929 01:04:47.320366 14769 consensus_queue.cc:260] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } }
I20260929 01:04:47.321321 14769 ts_tablet_manager.cc:1428] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 01:04:47.323943 14682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38339
I20260929 01:04:47.326301 14877 heartbeater.cc:503] Master 127.0.98.62:41481 was elected leader, sending a full tablet report...
Sep 29 01:04:47 dist-test-slave-7zdl krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 01:04:47.334364 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:47.323126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35584 (local address 127.0.98.1:33439)
0929 01:04:47.324703 (+ 1577us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:47.324712 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:47.324730 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:04:47.324784 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:47.324787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:47.324975 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:47.325104 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:47.325830 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.326858 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.328038 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.328362 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.328444 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:47.330758 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:47.330767 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:47.330816 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:47.330846 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:47.332512 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:47.332997 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:47.333002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:47.333004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:47.333090 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:47.333367 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:47.333372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:47.333374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:47.333604 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:47.333751 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:47.334068 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 01:04:47.334220 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1486,"thread_start_us":82,"threads_started":1}
I20260929 01:04:47.334380 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:47.323056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33439 (local address 127.0.0.1:35584)
0929 01:04:47.323472 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:47.323519 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:47.323643 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:47.325132 (+ 1489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:47.325139 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:47.325558 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:47.325687 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:47.325694 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.326991 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.326994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:47.327925 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:47.327936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.328461 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.328477 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:47.328583 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:47.329116 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:47.329131 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:47.330629 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:47.332645 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:47.332653 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:47.332658 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:47.332883 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:47.333188 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:47.333191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:47.333193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:47.333279 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:47.333727 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:47.333733 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:47.333856 (+ 123us) client_negotiation.cc:769] Sending connection context
0929 01:04:47.333992 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 01:04:47.334187 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":130,"threads_started":1}
I20260929 01:04:47.335387 14811 tablet_service.cc:1514] Processing DeleteTablet for tablet e9b40ccc554e4c3795c48a0e44371896 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:04:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35584
I20260929 01:04:47.336320 14886 tablet_replica.cc:330] stopping tablet replica
I20260929 01:04:47.336611 14886 raft_consensus.cc:2229] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:04:47.336894 14886 raft_consensus.cc:2258] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:04:47.337865 14886 ts_tablet_manager.cc:1905] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:04:47.341405 14886 ts_tablet_manager.cc:1918] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:04:47.341513 14886 log.cc:1196] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/wal/wals/e9b40ccc554e4c3795c48a0e44371896
I20260929 01:04:47.341782 14886 ts_tablet_manager.cc:1939] T e9b40ccc554e4c3795c48a0e44371896 P aee98792440b4cad86a9757cd709a314: Deleting consensus metadata
I20260929 01:04:47.343366 14673 catalog_manager.cc:4841] TS aee98792440b4cad86a9757cd709a314 (127.0.98.1:33439): tablet e9b40ccc554e4c3795c48a0e44371896 (table test-table [id=dfe2d1e0e81848ba92bcd8dd7cd7f248]) successfully deleted
W20260929 01:04:47.418030 14881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:47.418272 14881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:47.418313 14881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:47.422327 14881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:47.422379 14881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:47.422433 14881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:04:47.427567 14881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46085
--tserver_master_addrs=127.0.98.62:41481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:04:47.428483 14881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:47.429494 14881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:47.438692 14892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:47.438692 14894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:47.438906 14891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:47.439072 14881 server_base.cc:1034] running on GCE node
I20260929 01:04:47.439476 14881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:47.440058 14881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:47.441215 14881 hybrid_clock.cc:648] HybridClock initialized: now 1790643887441201 us; error 26 us; skew 500 ppm
Sep 29 01:04:47 dist-test-slave-7zdl krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:47.443861 14881 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:04:47.444594 14881 webserver.cc:460] Webserver started at http://127.0.98.2:46085/ using document root <none> and password file <none>
I20260929 01:04:47.445065 14881 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:47.445128 14881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:47.448737 14881 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:04:47.450577 14901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:47.451867 14881 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:04:47.451977 14881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "80ebf772f86d4df396ff05b6b433e785"
format_stamp: "Formatted at 2026-09-29 01:04:46 on dist-test-slave-7zdl"
server_key: "cd3687598f92a752ab0d32b49adac660"
server_key_iv: "421e3f2eae0d3b2a5776698d71a7b2c0"
server_key_version: "encryptionkey@0"
I20260929 01:04:47.452500 14881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:47.464187 14881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:47.467492 14881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:47.467700 14881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:47.468278 14881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:47.469115 14881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:47.469165 14881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:47.469244 14881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:47.469287 14881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:47.478440 14881 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37059
I20260929 01:04:47.478457 15014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37059 every 8 connection(s)
I20260929 01:04:47.479403 14881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 01:04:47 dist-test-slave-7zdl krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:04:47.488987 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 14881
I20260929 01:04:47.489392 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33801
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41481
--webserver_port=40517
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:04:47.492652 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:47.481122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44213 (local address 127.0.98.62:41481)
0929 01:04:47.481265 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:47.481268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:47.482476 (+ 1208us) server_negotiation.cc:407] Connection header received
0929 01:04:47.483407 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:47.483410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:47.483474 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:47.483579 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:47.484671 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.485201 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.486338 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.486577 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.486690 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:47.489057 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:47.489061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:47.489066 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:47.489120 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:47.490571 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:47.491138 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:47.491143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:47.491145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:47.491229 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:47.491568 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:47.491571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:47.491573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:47.491756 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:47.491880 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:47.492349 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 01:04:47.492486 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:04:47.493362 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:47.481453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.2:44213)
0929 01:04:47.482330 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:47.482369 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:47.483170 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:47.483755 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:47.483761 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:47.484085 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:47.484519 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:47.484531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.485343 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.485346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:47.486207 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:47.486216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.486723 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.486727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:47.486942 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:47.487506 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:47.487521 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:47.488891 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:47.490699 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:47.490705 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:47.490717 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:47.491037 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:47.491322 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:47.491326 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:47.491328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:47.491472 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:47.491878 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:47.491885 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:47.492111 (+ 226us) client_negotiation.cc:769] Sending connection context
0929 01:04:47.492290 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 01:04:47.492597 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":562,"threads_started":1}
I20260929 01:04:47.494486 15015 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41481
I20260929 01:04:47.494742 15015 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:47.495368 15015 heartbeater.cc:511] Master 127.0.98.62:41481 requested a full tablet report, sending...
I20260929 01:04:47.496508 14682 ts_manager.cc:194] Registered new tserver with Master: 80ebf772f86d4df396ff05b6b433e785 (127.0.98.2:37059)
I20260929 01:04:47.497398 14682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44213
W20260929 01:04:47.597886 15018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:47.598111 15018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:47.598151 15018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:47.601949 15018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:47.601994 15018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:47.602049 15018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:04:47.607018 15018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40517
--tserver_master_addrs=127.0.98.62:41481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:04:47.607894 15018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:47.608788 15018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:47.618518 15026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:47.618479 15024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:47.618742 15018 server_base.cc:1034] running on GCE node
W20260929 01:04:47.618475 15023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:47.619215 15018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:47.619851 15018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:47.621030 15018 hybrid_clock.cc:648] HybridClock initialized: now 1790643887620999 us; error 42 us; skew 500 ppm
Sep 29 01:04:47 dist-test-slave-7zdl krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:47.623749 15018 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:04:47.624575 15018 webserver.cc:460] Webserver started at http://127.0.98.3:40517/ using document root <none> and password file <none>
I20260929 01:04:47.625118 15018 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:47.625221 15018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:47.628937 15018 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 01:04:47.631134 15033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:47.632405 15018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 01:04:47.632524 15018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "50409a27acc24ff0b455aa410bfd076d"
format_stamp: "Formatted at 2026-09-29 01:04:46 on dist-test-slave-7zdl"
server_key: "3c7422f16c4a053f331f0634b3cdda50"
server_key_iv: "34f4b76f7c6b5446ad98a32ec391517b"
server_key_version: "encryptionkey@0"
I20260929 01:04:47.633031 15018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:47.649727 15018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:47.652858 15018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:47.653050 15018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:47.653787 15018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:47.654691 15018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:47.654769 15018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:47.654870 15018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:47.654917 15018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:47.664296 15018 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33801
I20260929 01:04:47.664325 15146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33801 every 8 connection(s)
I20260929 01:04:47.665305 15018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
Sep 29 01:04:47 dist-test-slave-7zdl krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:04:47.675484 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 15018
I20260929 01:04:47.678911 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:47.667406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34775 (local address 127.0.98.62:41481)
0929 01:04:47.667572 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:47.667576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:47.668546 (+ 970us) server_negotiation.cc:407] Connection header received
0929 01:04:47.669413 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:47.669417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:47.669475 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:47.669569 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:47.670726 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.671316 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.672453 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.672681 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.672806 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:47.675326 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:47.675330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:47.675332 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:47.675363 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:47.676846 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:47.677408 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:47.677413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:47.677414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:47.677493 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:47.677836 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:47.677839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:47.677841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:47.678014 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:47.678117 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:47.678607 (+ 490us) server_negotiation.cc:299] Negotiation successful
0929 01:04:47.678719 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 01:04:47.679714 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:47.667780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.3:34775)
0929 01:04:47.668406 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:47.668445 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:47.669215 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:47.669746 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:47.669752 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:47.670124 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:47.670578 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:47.670591 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.671465 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.671468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:47.672334 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:47.672342 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.672844 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.672847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:47.673025 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:47.673635 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:47.673651 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:47.675129 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:47.676965 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:47.676971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:47.676984 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:47.677321 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:47.677587 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:47.677590 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:47.677592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:47.677734 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:47.678114 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:47.678121 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:47.678354 (+ 233us) client_negotiation.cc:769] Sending connection context
0929 01:04:47.678567 (+ 213us) client_negotiation.cc:240] Negotiation successful
0929 01:04:47.678917 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":112,"threads_started":1}
I20260929 01:04:47.680863 15147 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41481
I20260929 01:04:47.681129 15147 heartbeater.cc:464] Registering TS with master...
I20260929 01:04:47.681700 15147 heartbeater.cc:511] Master 127.0.98.62:41481 requested a full tablet report, sending...
I20260929 01:04:47.682885 14682 ts_manager.cc:194] Registered new tserver with Master: 50409a27acc24ff0b455aa410bfd076d (127.0.98.3:33801)
I20260929 01:04:47.683570 14682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34775
I20260929 01:04:47.684603 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:47.676172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43774 (local address 127.0.98.62:41481)
0929 01:04:47.677059 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:47.677065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:47.677077 (+ 12us) server_negotiation.cc:407] Connection header received
0929 01:04:47.677108 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:47.677111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:47.677150 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:47.677254 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:47.677983 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.678517 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.679557 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.679834 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.679940 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:47.681361 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:47.681364 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:47.681366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:47.681386 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:47.682966 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:47.683368 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:47.683371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:47.683373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:47.683453 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:47.683705 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:47.683709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:47.683711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:47.683933 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:47.684136 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:47.684310 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 01:04:47.684424 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":805,"thread_start_us":97,"threads_started":1}
I20260929 01:04:47.686380 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:04:47.690604 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:47.687178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43778 (local address 127.0.98.62:41481)
0929 01:04:47.687339 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:47.687393 (+ 54us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:47.687505 (+ 112us) server_negotiation.cc:407] Connection header received
0929 01:04:47.687690 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:47.687693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:47.687773 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:47.687851 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:04:47.688119 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.688570 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.689393 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.689594 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.689670 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:47.689927 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:04:47.690254 (+ 327us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:04:47.690336 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:47.690421 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 01:04:47.690461 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 01:04:47.691285 14682 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:04:47.694345 14682 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:04:47.702366 14811 tablet_service.cc:1467] Processing CreateTablet for tablet 31f9ed0c9b11424fb0b3ac8d1d2253e1 (DEFAULT_TABLE table=test-table [id=01cd1b81acd048fe94e22f36680491b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:04:47.703122 14811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f9ed0c9b11424fb0b3ac8d1d2253e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:47.705979 14769 tablet_bootstrap.cc:492] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314: Bootstrap starting.
I20260929 01:04:47.706872 14769 tablet_bootstrap.cc:654] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:47.707978 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:47.701465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33801 (local address 127.0.0.1:38302)
0929 01:04:47.701876 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:47.701889 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:47.701972 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:47.703166 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:47.703169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:47.703190 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:47.703306 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:47.703312 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.704645 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.704648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:47.705948 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:47.705958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.707476 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.707479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:47.707598 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:47.707646 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 01:04:47.707726 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 01:04:47.707817 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":1,"thread_start_us":76,"threads_started":1}
I20260929 01:04:47.708606 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:47.701630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38302 (local address 127.0.98.3:33801)
0929 01:04:47.702499 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:47.702512 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:47.702535 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:04:47.702609 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:47.702615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:47.702858 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:47.703001 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:47.703429 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.704488 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.706146 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.707103 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.707202 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:47.707377 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:47.708282 (+ 905us) server_negotiation.cc:299] Negotiation successful
0929 01:04:47.708411 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":143,"threads_started":1}
I20260929 01:04:47.708619 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:47.701052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37059 (local address 127.0.0.1:60634)
0929 01:04:47.701165 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:47.701180 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:47.701275 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:47.702163 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:47.702166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:47.702183 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:47.702314 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:47.702324 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.703861 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.703867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:47.705139 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:47.705153 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.708080 (+ 2927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.708087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:47.708261 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:47.708291 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 01:04:47.708352 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 01:04:47.708454 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":48}
I20260929 01:04:47.709616 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:47.701277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60634 (local address 127.0.98.2:37059)
0929 01:04:47.701659 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:47.701672 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:47.701707 (+ 35us) server_negotiation.cc:407] Connection header received
0929 01:04:47.701793 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:47.701799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:47.702018 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:47.702187 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:47.702576 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.703733 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.706393 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.707848 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.708732 (+ 884us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:47.709264 (+ 532us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:47.709320 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 01:04:47.709441 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":163,"threads_started":1}
I20260929 01:04:47.709834 14769 tablet_bootstrap.cc:492] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314: No bootstrap required, opened a new log
I20260929 01:04:47.710223 14769 ts_tablet_manager.cc:1397] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260929 01:04:47.710724 14769 raft_consensus.cc:348] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } }
I20260929 01:04:47.710860 14769 raft_consensus.cc:374] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:47.710901 14769 raft_consensus.cc:729] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee98792440b4cad86a9757cd709a314, State: Initialized, Role: FOLLOWER
I20260929 01:04:47.711027 14769 consensus_queue.cc:260] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } }
I20260929 01:04:47.712846 14949 tablet_service.cc:1467] Processing CreateTablet for tablet 31f9ed0c9b11424fb0b3ac8d1d2253e1 (DEFAULT_TABLE table=test-table [id=01cd1b81acd048fe94e22f36680491b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:04:47.713207 15081 tablet_service.cc:1467] Processing CreateTablet for tablet 31f9ed0c9b11424fb0b3ac8d1d2253e1 (DEFAULT_TABLE table=test-table [id=01cd1b81acd048fe94e22f36680491b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:04:47.713936 14949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f9ed0c9b11424fb0b3ac8d1d2253e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:47.714118 15081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f9ed0c9b11424fb0b3ac8d1d2253e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:04:47.711298 14769 ts_tablet_manager.cc:1428] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 01:04:47.722092 15161 tablet_bootstrap.cc:492] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d: Bootstrap starting.
I20260929 01:04:47.722092 15160 tablet_bootstrap.cc:492] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 80ebf772f86d4df396ff05b6b433e785: Bootstrap starting.
I20260929 01:04:47.724491 15161 tablet_bootstrap.cc:654] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:47.724931 15160 tablet_bootstrap.cc:654] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 80ebf772f86d4df396ff05b6b433e785: Neither blocks nor log segments found. Creating new log.
I20260929 01:04:47.725819 15161 log.cc:824] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:47.725819 15160 log.cc:824] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 80ebf772f86d4df396ff05b6b433e785: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:47.727849 15161 tablet_bootstrap.cc:492] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d: No bootstrap required, opened a new log
I20260929 01:04:47.727923 15160 tablet_bootstrap.cc:492] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 80ebf772f86d4df396ff05b6b433e785: No bootstrap required, opened a new log
I20260929 01:04:47.728046 15161 ts_tablet_manager.cc:1397] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 01:04:47.728061 15160 ts_tablet_manager.cc:1397] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 80ebf772f86d4df396ff05b6b433e785: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 01:04:47.732050 15161 raft_consensus.cc:348] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } }
I20260929 01:04:47.732050 15160 raft_consensus.cc:348] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 80ebf772f86d4df396ff05b6b433e785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } }
I20260929 01:04:47.732339 15160 raft_consensus.cc:374] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 80ebf772f86d4df396ff05b6b433e785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:47.732340 15161 raft_consensus.cc:374] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:04:47.732403 15161 raft_consensus.cc:729] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50409a27acc24ff0b455aa410bfd076d, State: Initialized, Role: FOLLOWER
I20260929 01:04:47.732410 15160 raft_consensus.cc:729] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 80ebf772f86d4df396ff05b6b433e785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80ebf772f86d4df396ff05b6b433e785, State: Initialized, Role: FOLLOWER
I20260929 01:04:47.732882 15161 consensus_queue.cc:260] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } }
I20260929 01:04:47.732882 15160 consensus_queue.cc:260] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 80ebf772f86d4df396ff05b6b433e785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } }
I20260929 01:04:47.733542 15015 heartbeater.cc:503] Master 127.0.98.62:41481 was elected leader, sending a full tablet report...
I20260929 01:04:47.733862 15147 heartbeater.cc:503] Master 127.0.98.62:41481 was elected leader, sending a full tablet report...
I20260929 01:04:47.733898 15161 ts_tablet_manager.cc:1428] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 01:04:47.734881 15160 ts_tablet_manager.cc:1428] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 80ebf772f86d4df396ff05b6b433e785: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260929 01:04:47.798626 14878 tablet.cc:2369] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:04:47.883692 15165 raft_consensus.cc:482] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:04:47.883949 15165 raft_consensus.cc:504] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } }
I20260929 01:04:47.885142 15165 leader_election.cc:288] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80ebf772f86d4df396ff05b6b433e785 (127.0.98.2:37059), aee98792440b4cad86a9757cd709a314 (127.0.98.1:33439)
I20260929 01:04:47.889588 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:47.885489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55455 (local address 127.0.98.2:37059)
0929 01:04:47.885626 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:47.885632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:47.885928 (+ 296us) server_negotiation.cc:407] Connection header received
0929 01:04:47.886081 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:47.886085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:47.886151 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:47.886243 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:47.886896 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.887469 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.888614 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.889022 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.889135 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:47.889199 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:47.889372 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 01:04:47.889458 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 01:04:47.889590 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:47.885409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33439 (local address 127.0.98.3:49451)
0929 01:04:47.885525 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:47.885544 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:47.885648 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:47.886399 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:47.886402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:47.886419 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:47.886517 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:47.886522 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.887385 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.887388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:47.888429 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:47.888439 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.889164 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.889168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:47.889292 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:47.889328 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 01:04:47.889408 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 01:04:47.889470 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":19}
I20260929 01:04:47.889607 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:47.885411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37059 (local address 127.0.98.3:55455)
0929 01:04:47.885814 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:47.885829 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:47.885940 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:47.886267 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:47.886270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:47.886286 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:04:47.886759 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:47.886767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.887587 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.887589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:47.888463 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:47.888472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.889146 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.889150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:47.889239 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:47.889261 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 01:04:47.889337 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 01:04:47.889415 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":59,"thread_start_us":112,"threads_started":1}
I20260929 01:04:47.889624 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:47.885747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49451 (local address 127.0.98.1:33439)
0929 01:04:47.886108 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:47.886113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:47.886127 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:04:47.886191 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:47.886193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:47.886278 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:47.886389 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:04:47.886634 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.887274 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.888553 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.889025 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.889149 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:47.889333 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:47.889429 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 01:04:47.889507 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":119,"threads_started":1}
I20260929 01:04:47.890363 14831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f9ed0c9b11424fb0b3ac8d1d2253e1" candidate_uuid: "50409a27acc24ff0b455aa410bfd076d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee98792440b4cad86a9757cd709a314" is_pre_election: true
I20260929 01:04:47.890457 14969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f9ed0c9b11424fb0b3ac8d1d2253e1" candidate_uuid: "50409a27acc24ff0b455aa410bfd076d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80ebf772f86d4df396ff05b6b433e785" is_pre_election: true
I20260929 01:04:47.890697 14831 raft_consensus.cc:2454] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50409a27acc24ff0b455aa410bfd076d in term 0.
I20260929 01:04:47.890762 14969 raft_consensus.cc:2454] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 80ebf772f86d4df396ff05b6b433e785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50409a27acc24ff0b455aa410bfd076d in term 0.
I20260929 01:04:47.891203 15034 leader_election.cc:302] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50409a27acc24ff0b455aa410bfd076d, aee98792440b4cad86a9757cd709a314; no voters:
I20260929 01:04:47.891497 15165 raft_consensus.cc:2781] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:04:47.891582 15165 raft_consensus.cc:482] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:04:47.891624 15165 raft_consensus.cc:3037] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:47.892704 15165 raft_consensus.cc:504] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } }
I20260929 01:04:47.893054 15165 leader_election.cc:288] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d [CANDIDATE]: Term 1 election: Requested vote from peers 80ebf772f86d4df396ff05b6b433e785 (127.0.98.2:37059), aee98792440b4cad86a9757cd709a314 (127.0.98.1:33439)
I20260929 01:04:47.893374 14969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f9ed0c9b11424fb0b3ac8d1d2253e1" candidate_uuid: "50409a27acc24ff0b455aa410bfd076d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80ebf772f86d4df396ff05b6b433e785"
I20260929 01:04:47.893417 14831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f9ed0c9b11424fb0b3ac8d1d2253e1" candidate_uuid: "50409a27acc24ff0b455aa410bfd076d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee98792440b4cad86a9757cd709a314"
I20260929 01:04:47.893514 14831 raft_consensus.cc:3037] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:47.893508 14969 raft_consensus.cc:3037] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 80ebf772f86d4df396ff05b6b433e785 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:04:47.894412 14831 raft_consensus.cc:2454] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50409a27acc24ff0b455aa410bfd076d in term 1.
I20260929 01:04:47.894603 14969 raft_consensus.cc:2454] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 80ebf772f86d4df396ff05b6b433e785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50409a27acc24ff0b455aa410bfd076d in term 1.
I20260929 01:04:47.894672 15034 leader_election.cc:302] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50409a27acc24ff0b455aa410bfd076d, aee98792440b4cad86a9757cd709a314; no voters:
I20260929 01:04:47.894867 15165 raft_consensus.cc:2781] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:04:47.895092 15165 raft_consensus.cc:686] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d [term 1 LEADER]: Becoming Leader. State: Replica: 50409a27acc24ff0b455aa410bfd076d, State: Running, Role: LEADER
I20260929 01:04:47.895443 15165 consensus_queue.cc:237] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } }
I20260929 01:04:47.898630 14681 catalog_manager.cc:5495] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d reported cstate change: term changed from 0 to 1, leader changed from <none> to 50409a27acc24ff0b455aa410bfd076d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "50409a27acc24ff0b455aa410bfd076d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } health_report { overall_health: UNKNOWN } } }
W20260929 01:04:47.916253 15148 tablet.cc:2369] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:04:47.940724 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:47.937216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38312 (local address 127.0.98.3:33801)
0929 01:04:47.937351 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:47.937355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:47.937367 (+ 12us) server_negotiation.cc:407] Connection header received
0929 01:04:47.937438 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:47.937441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:47.937495 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:47.937585 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:04:47.937822 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.938356 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.939264 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:47.939505 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:47.939590 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:47.939942 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:04:47.940363 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:04:47.940454 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:47.940549 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 01:04:47.940600 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 01:04:47.946359 14831 raft_consensus.cc:1261] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [term 1 FOLLOWER]: Refusing update from remote peer 50409a27acc24ff0b455aa410bfd076d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:04:47.946363 14969 raft_consensus.cc:1261] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 80ebf772f86d4df396ff05b6b433e785 [term 1 FOLLOWER]: Refusing update from remote peer 50409a27acc24ff0b455aa410bfd076d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:04:47.947039 15168 consensus_queue.cc:1035] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d [LEADER]: Connected to new peer: Peer: permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:04:47.947187 15165 consensus_queue.cc:1035] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d [LEADER]: Connected to new peer: Peer: permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:04:47.954532 15172 mvcc.cc:205] Tried to move back new op lower bound from 7334477365021384704 to 7334477364822065152. Current Snapshot: MvccSnapshot[applied={T|T < 7334477365021384704}]
I20260929 01:04:47.961390 14681 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:04:47.961521 14681 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:04:47.963748 14681 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e: Sending DeleteTablet for 3 replicas of tablet 31f9ed0c9b11424fb0b3ac8d1d2253e1
I20260929 01:04:47.964365 14949 tablet_service.cc:1514] Processing DeleteTablet for tablet 31f9ed0c9b11424fb0b3ac8d1d2253e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:04:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60634
I20260929 01:04:47.964469 15081 tablet_service.cc:1514] Processing DeleteTablet for tablet 31f9ed0c9b11424fb0b3ac8d1d2253e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:04:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38302
I20260929 01:04:47.964834 15180 tablet_replica.cc:330] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 80ebf772f86d4df396ff05b6b433e785: stopping tablet replica
I20260929 01:04:47.964907 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 14727
I20260929 01:04:47.964902 14811 tablet_service.cc:1514] Processing DeleteTablet for tablet 31f9ed0c9b11424fb0b3ac8d1d2253e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:04:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35584
I20260929 01:04:47.965133 15180 raft_consensus.cc:2229] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 80ebf772f86d4df396ff05b6b433e785 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:04:47.965586 15181 tablet_replica.cc:330] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d: stopping tablet replica
I20260929 01:04:47.965621 15180 raft_consensus.cc:2258] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 80ebf772f86d4df396ff05b6b433e785 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:04:47.965909 15181 raft_consensus.cc:2229] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:04:47.966629 15181 raft_consensus.cc:2258] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:04:47.967239 15180 ts_tablet_manager.cc:1905] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 80ebf772f86d4df396ff05b6b433e785: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:04:47.967890 15181 ts_tablet_manager.cc:1905] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:04:47.970549 15181 ts_tablet_manager.cc:1918] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:04:47.970688 15181 log.cc:1196] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/31f9ed0c9b11424fb0b3ac8d1d2253e1
I20260929 01:04:47.970738 15180 ts_tablet_manager.cc:1918] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 80ebf772f86d4df396ff05b6b433e785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:04:47.970947 15180 log.cc:1196] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 80ebf772f86d4df396ff05b6b433e785: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/wal/wals/31f9ed0c9b11424fb0b3ac8d1d2253e1
I20260929 01:04:47.971364 15181 ts_tablet_manager.cc:1939] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 50409a27acc24ff0b455aa410bfd076d: Deleting consensus metadata
I20260929 01:04:47.971414 15180 ts_tablet_manager.cc:1939] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P 80ebf772f86d4df396ff05b6b433e785: Deleting consensus metadata
I20260929 01:04:47.972497 14673 catalog_manager.cc:4841] TS 50409a27acc24ff0b455aa410bfd076d (127.0.98.3:33801): tablet 31f9ed0c9b11424fb0b3ac8d1d2253e1 (table test-table [id=01cd1b81acd048fe94e22f36680491b3]) successfully deleted
I20260929 01:04:47.972675 14674 catalog_manager.cc:4841] TS 80ebf772f86d4df396ff05b6b433e785 (127.0.98.2:37059): tablet 31f9ed0c9b11424fb0b3ac8d1d2253e1 (table test-table [id=01cd1b81acd048fe94e22f36680491b3]) successfully deleted
W20260929 01:04:47.973470 14673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33439 (error 108)
I20260929 01:04:47.973815 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 14881
I20260929 01:04:47.975229 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:47.974876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33439 (local address 127.0.0.1:35598)
0929 01:04:47.975054 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:47.975130 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260929 01:04:47.975443 14673 catalog_manager.cc:4584] TS aee98792440b4cad86a9757cd709a314 (127.0.98.1:33439): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 31f9ed0c9b11424fb0b3ac8d1d2253e1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33439: connect: Connection refused (error 111)
I20260929 01:04:47.978946 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 15018
I20260929 01:04:47.984905 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 14656
I20260929 01:04:47.991292 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41481
--webserver_interface=127.0.98.62
--webserver_port=41195
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:04:48.097975 15183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:48.098268 15183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:48.098347 15183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:48.102311 15183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:04:48.102412 15183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:48.102452 15183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:48.102507 15183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:04:48.102543 15183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:04:48.107712 15183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:04:48.108753 15183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:48.109735 15183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:48.118158 15188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:48.118158 15191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:48.118194 15189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:48.118336 15183 server_base.cc:1034] running on GCE node
I20260929 01:04:48.119134 15183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:48.120260 15183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:48.121445 15183 hybrid_clock.cc:648] HybridClock initialized: now 1790643888121416 us; error 53 us; skew 500 ppm
Sep 29 01:04:48 dist-test-slave-7zdl krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:48.124109 15183 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:04:48.124935 15183 webserver.cc:460] Webserver started at http://127.0.98.62:41195/ using document root <none> and password file <none>
I20260929 01:04:48.125492 15183 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:48.125583 15183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:48.129246 15183 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:04:48.131268 15198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:48.132449 15183 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:04:48.132587 15183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "72a783dac216456a9482b052f226ff5e"
format_stamp: "Formatted at 2026-09-29 01:04:45 on dist-test-slave-7zdl"
server_key: "13e1231a8fe278fc6d740d0415c80d3f"
server_key_iv: "8ce6e14e470b131a752cf61f97d0b3f4"
server_key_version: "encryptionkey@0"
I20260929 01:04:48.133107 15183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:48.142148 15183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:48.144820 15183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:48.145028 15183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:48.152604 15183 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41481
I20260929 01:04:48.152887 15250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41481 every 8 connection(s)
I20260929 01:04:48.153661 15183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:04:48.156921 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 15183
I20260929 01:04:48.157330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33439
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41481
--webserver_port=35165
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:04:48.157986 15251 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:04:48.159637 15251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e: Bootstrap starting.
I20260929 01:04:48.176679 15251 log.cc:824] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:48.187773 15251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:04:48.188148 15251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e: Bootstrap complete.
I20260929 01:04:48.192178 15251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a783dac216456a9482b052f226ff5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41481 } }
I20260929 01:04:48.192531 15251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72a783dac216456a9482b052f226ff5e, State: Initialized, Role: FOLLOWER
I20260929 01:04:48.192993 15251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a783dac216456a9482b052f226ff5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41481 } }
I20260929 01:04:48.193105 15251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:04:48.193197 15251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:04:48.193308 15251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:04:48.195252 15251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a783dac216456a9482b052f226ff5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41481 } }
I20260929 01:04:48.195571 15251 leader_election.cc:302] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72a783dac216456a9482b052f226ff5e; no voters:
I20260929 01:04:48.195894 15251 leader_election.cc:288] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 01:04:48.195968 15255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [term 3 FOLLOWER]: Leader election won for term 3
I20260929 01:04:48.196146 15255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [term 3 LEADER]: Becoming Leader. State: Replica: 72a783dac216456a9482b052f226ff5e, State: Running, Role: LEADER
I20260929 01:04:48.196430 15255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a783dac216456a9482b052f226ff5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41481 } }
I20260929 01:04:48.197068 15251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:04:48.197389 15256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72a783dac216456a9482b052f226ff5e. Latest consensus state: current_term: 3 leader_uuid: "72a783dac216456a9482b052f226ff5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a783dac216456a9482b052f226ff5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41481 } } }
I20260929 01:04:48.197400 15257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "72a783dac216456a9482b052f226ff5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a783dac216456a9482b052f226ff5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41481 } } }
I20260929 01:04:48.197525 15256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:48.197552 15257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72a783dac216456a9482b052f226ff5e [sys.catalog]: This master's current role is: LEADER
I20260929 01:04:48.197970 15259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:04:48.203080 15259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:04:48.203744 15259 catalog_manager.cc:1259] Loaded cluster ID: 8f4804fedb6848e99acedd943c9d443d
I20260929 01:04:48.203806 15259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 01:04:48.204773 15259 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4fbee229ad google::LogMessage::Fail() at ??:0
@ 0x7f4fbee2588a google::LogMessage::SendToLog() at ??:0
@ 0x7f4fbee224de google::LogMessage::Flush() at ??:0
@ 0x7f4fbee23a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4fc9761fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f4fc9762d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4fc975f579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f4fc97a3640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4fc10743d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4fc1074c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4fc10763f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4fc1065512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4fc26a96db start_thread at ??:0
@ 0x7f4fbd92061f clone at ??:0
W20260929 01:04:48.293241 15253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:48.293610 15253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:48.293687 15253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:48.299999 15253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:48.300074 15253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:48.300168 15253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:04:48.308077 15253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35165
--tserver_master_addrs=127.0.98.62:41481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:04:48.309378 15253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:48.310671 15253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:48.321502 15293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:48.321632 15292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:48.321514 15295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:48.321878 15253 server_base.cc:1034] running on GCE node
I20260929 01:04:48.322325 15253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:48.323112 15253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:48.324306 15253 hybrid_clock.cc:648] HybridClock initialized: now 1790643888324270 us; error 47 us; skew 500 ppm
Sep 29 01:04:48 dist-test-slave-7zdl krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:48.327322 15253 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:04:48.328143 15253 webserver.cc:460] Webserver started at http://127.0.98.1:35165/ using document root <none> and password file <none>
I20260929 01:04:48.328702 15253 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:48.328769 15253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:48.332718 15253 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:04:48.335170 15302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:48.336445 15253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:04:48.336589 15253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "aee98792440b4cad86a9757cd709a314"
format_stamp: "Formatted at 2026-09-29 01:04:45 on dist-test-slave-7zdl"
server_key: "be75733d1e65f0811a754e23dbfb589c"
server_key_iv: "e0696b1333a782b9be0e58eb3f3804ba"
server_key_version: "encryptionkey@0"
I20260929 01:04:48.337112 15253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:48.346910 15253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:48.350450 15253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:48.350642 15253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:48.351351 15253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:48.352689 15309 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 01:04:48.356925 15253 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 01:04:48.356997 15253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260929 01:04:48.357061 15253 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 01:04:48.359239 15253 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 01:04:48.359297 15253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260929 01:04:48.359581 15309 tablet_bootstrap.cc:492] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314: Bootstrap starting.
I20260929 01:04:48.369858 15253 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33439
I20260929 01:04:48.369886 15416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33439 every 8 connection(s)
I20260929 01:04:48.371057 15253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:04:48.373766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 15253
I20260929 01:04:48.374321 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37059
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41481
--webserver_port=46085
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:04:48.379459 15309 log.cc:824] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314: Log is configured to *not* fsync() on all Append() calls
I20260929 01:04:48.389336 15309 tablet_bootstrap.cc:492] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:04:48.389734 15309 tablet_bootstrap.cc:492] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314: Bootstrap complete.
I20260929 01:04:48.391156 15309 ts_tablet_manager.cc:1397] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314: Time spent bootstrapping tablet: real 0.032s user 0.020s sys 0.010s
I20260929 01:04:48.394167 15309 raft_consensus.cc:348] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } }
I20260929 01:04:48.394747 15309 raft_consensus.cc:729] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee98792440b4cad86a9757cd709a314, State: Initialized, Role: FOLLOWER
I20260929 01:04:48.395285 15309 consensus_queue.cc:260] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } }
I20260929 01:04:48.396286 15309 ts_tablet_manager.cc:1428] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
W20260929 01:04:48.442444 15419 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:04:48.373421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.1:37757)
0929 01:04:48.373914 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:48.373967 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:48.375024 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:48.441705 (+ 66681us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":143,"threads_started":1}
W20260929 01:04:48.443042 15417 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41481 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41481: Client connection negotiation failed: client connection to 127.0.98.62:41481: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:04:48.443789 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:48.443524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.1:40939)
0929 01:04:48.443652 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:48.443725 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 01:04:48.444466 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:48.444263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.1:48501)
0929 01:04:48.444363 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:48.444407 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 01:04:48.444703 15417 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 01:04:48.490463 15420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:48.490697 15420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:48.490739 15420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:48.494719 15420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:48.494787 15420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:48.494858 15420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:04:48.499781 15420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46085
--tserver_master_addrs=127.0.98.62:41481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:04:48.500669 15420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:48.501643 15420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:48.510775 15427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:48.510784 15420 server_base.cc:1034] running on GCE node
W20260929 01:04:48.510747 15430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:48.510754 15428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:48.511397 15420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:48.511982 15420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:48.513139 15420 hybrid_clock.cc:648] HybridClock initialized: now 1790643888513125 us; error 24 us; skew 500 ppm
Sep 29 01:04:48 dist-test-slave-7zdl krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:48.515834 15420 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:04:48.516651 15420 webserver.cc:460] Webserver started at http://127.0.98.2:46085/ using document root <none> and password file <none>
I20260929 01:04:48.517191 15420 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:48.517249 15420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:48.521087 15420 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20260929 01:04:48.523072 15437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:48.524242 15420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 01:04:48.524370 15420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "80ebf772f86d4df396ff05b6b433e785"
format_stamp: "Formatted at 2026-09-29 01:04:46 on dist-test-slave-7zdl"
server_key: "cd3687598f92a752ab0d32b49adac660"
server_key_iv: "421e3f2eae0d3b2a5776698d71a7b2c0"
server_key_version: "encryptionkey@0"
I20260929 01:04:48.524880 15420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:48.543361 15420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:48.545971 15420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:48.546165 15420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:48.546721 15420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:48.547639 15420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:48.547712 15420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:48.547782 15420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:48.547835 15420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:48.556895 15420 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37059
I20260929 01:04:48.556924 15550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37059 every 8 connection(s)
I20260929 01:04:48.557835 15420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:04:48.561244 15553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:48.559830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.2:33003)
0929 01:04:48.560468 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:48.560569 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":121,"threads_started":1}
W20260929 01:04:48.561717 15551 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41481 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41481: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
I20260929 01:04:48.562264 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 15420
I20260929 01:04:48.562356 15553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:48.562116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.2:56157)
0929 01:04:48.562219 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:48.562266 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 01:04:48.562680 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33801
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41481
--webserver_port=40517
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:04:48.563031 15553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:48.562775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.2:55493)
0929 01:04:48.562939 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:48.562986 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260929 01:04:48.563257 15551 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 01:04:48.672170 15554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:04:48.672428 15554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:04:48.672467 15554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:04:48.676589 15554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:04:48.676642 15554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:04:48.676726 15554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:04:48.682129 15554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40517
--tserver_master_addrs=127.0.98.62:41481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:04:48.683234 15554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:04:48.684274 15554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:04:48.694020 15562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:04:48.694016 15560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:48.694257 15554 server_base.cc:1034] running on GCE node
W20260929 01:04:48.694044 15559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:04:48.694716 15554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:04:48.695415 15554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:04:48.696578 15554 hybrid_clock.cc:648] HybridClock initialized: now 1790643888696558 us; error 32 us; skew 500 ppm
Sep 29 01:04:48 dist-test-slave-7zdl krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:04:48.699343 15554 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:04:48.700170 15554 webserver.cc:460] Webserver started at http://127.0.98.3:40517/ using document root <none> and password file <none>
I20260929 01:04:48.700686 15554 fs_manager.cc:359] Metadata directory not provided
I20260929 01:04:48.700744 15554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:04:48.704450 15554 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:04:48.706358 15569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:48.707681 15554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:04:48.707800 15554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "50409a27acc24ff0b455aa410bfd076d"
format_stamp: "Formatted at 2026-09-29 01:04:46 on dist-test-slave-7zdl"
server_key: "3c7422f16c4a053f331f0634b3cdda50"
server_key_iv: "34f4b76f7c6b5446ad98a32ec391517b"
server_key_version: "encryptionkey@0"
I20260929 01:04:48.708324 15554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:04:48.717944 15554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:04:48.720831 15554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:04:48.721019 15554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:04:48.721575 15554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:04:48.722352 15554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:04:48.722395 15554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:48.722460 15554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:04:48.722497 15554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:04:48.731521 15554 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33801
I20260929 01:04:48.731612 15682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33801 every 8 connection(s)
I20260929 01:04:48.732547 15554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:04:48.736150 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:48.734580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.3:48783)
0929 01:04:48.735288 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:48.735394 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":121,"threads_started":1}
W20260929 01:04:48.736552 15683 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41481 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41481: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
I20260929 01:04:48.737114 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:48.736888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.3:50191)
0929 01:04:48.737000 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:48.737049 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 01:04:48.737658 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:48.737452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.3:36835)
0929 01:04:48.737558 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:48.737600 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 01:04:48.737820 15683 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 01:04:48.738618 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 15554
I20260929 01:04:49.445677 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:49.445272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.1:60533)
0929 01:04:49.445506 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:49.445566 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":72,"threads_started":1}
I20260929 01:04:49.564179 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:49.563785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.2:41869)
0929 01:04:49.564023 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:49.564072 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":93,"threads_started":1}
I20260929 01:04:49.738734 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:49.738345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.3:36813)
0929 01:04:49.738578 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:49.738628 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":63,"threads_started":1}
I20260929 01:04:49.946710 15690 raft_consensus.cc:482] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:04:49.946951 15690 raft_consensus.cc:504] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } }
I20260929 01:04:49.948099 15690 leader_election.cc:288] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 80ebf772f86d4df396ff05b6b433e785 (127.0.98.2:37059), 50409a27acc24ff0b455aa410bfd076d (127.0.98.3:33801)
Sep 29 01:04:49 dist-test-slave-7zdl krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 29 01:04:49 dist-test-slave-7zdl krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260929 01:04:49.961508 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:49.948348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37059 (local address 127.0.98.1:41505)
0929 01:04:49.948594 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:49.948608 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:49.948714 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:49.950408 (+ 1694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:49.950413 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:49.950985 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:49.951439 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:49.951452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:49.953018 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:49.953021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:49.953732 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:49.953740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:49.954449 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:49.954452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:49.954620 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:49.955342 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:49.955354 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:49.956893 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:49.959181 (+ 2288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:49.959189 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:49.959205 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:49.959528 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:49.959865 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:49.959871 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:49.959873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:49.960006 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:49.960586 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:49.960592 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:49.960843 (+ 251us) client_negotiation.cc:769] Sending connection context
0929 01:04:49.961017 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 01:04:49.961314 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":89,"spinlock_wait_cycles":14208,"thread_start_us":59,"threads_started":1}
I20260929 01:04:49.961548 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:49.948513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56737 (local address 127.0.98.3:33801)
0929 01:04:49.948966 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:49.948998 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:49.949041 (+ 43us) server_negotiation.cc:407] Connection header received
0929 01:04:49.949728 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:49.949734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:49.950061 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:49.950361 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:49.951597 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:49.952573 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:49.953879 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:49.954306 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:49.954466 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:49.957388 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:49.957393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:49.957403 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:49.957427 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:49.959162 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:49.959669 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:49.959673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:49.959677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:49.959784 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:49.960102 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:49.960106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:49.960108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:49.960468 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:49.960609 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:49.961132 (+ 523us) server_negotiation.cc:299] Negotiation successful
0929 01:04:49.961386 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":88,"threads_started":1}
I20260929 01:04:49.961848 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:49.948347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33801 (local address 127.0.98.1:56737)
0929 01:04:49.948698 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:49.948712 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:04:49.948837 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:04:49.950407 (+ 1570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:04:49.950412 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:04:49.950985 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:04:49.951439 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:49.951453 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:49.952710 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:49.952714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:49.953737 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:04:49.953746 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:04:49.954424 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:04:49.954427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:04:49.954622 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:49.955293 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:04:49.955309 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:04:49.957216 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:04:49.959280 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:49.959283 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:49.959284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:49.959542 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:49.959905 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:04:49.959907 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:04:49.959909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:04:49.960007 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:04:49.960587 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:04:49.960592 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:04:49.960841 (+ 249us) client_negotiation.cc:769] Sending connection context
0929 01:04:49.961017 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 01:04:49.961400 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":349,"spinlock_wait_cycles":10240,"thread_start_us":102,"threads_started":1}
I20260929 01:04:49.961589 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:49.948770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41505 (local address 127.0.98.2:37059)
0929 01:04:49.949081 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0929 01:04:49.949100 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 01:04:49.949130 (+ 30us) server_negotiation.cc:407] Connection header received
0929 01:04:49.949751 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:04:49.949756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:04:49.950062 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:04:49.950361 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:04:49.951602 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:49.952691 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:49.953844 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:04:49.954310 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:04:49.954480 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:04:49.957113 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:04:49.957117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:04:49.957128 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:04:49.957155 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:04:49.959075 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:49.959637 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:49.959645 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:49.959658 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:49.959769 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:04:49.960104 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:04:49.960108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:04:49.960109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:04:49.960468 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:04:49.960609 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 01:04:49.961136 (+ 527us) server_negotiation.cc:299] Negotiation successful
0929 01:04:49.961395 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":121,"threads_started":1}
W20260929 01:04:49.964156 15303 leader_election.cc:341] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 50409a27acc24ff0b455aa410bfd076d (127.0.98.3:33801): Not found: Tablet not found: 31f9ed0c9b11424fb0b3ac8d1d2253e1
W20260929 01:04:49.964377 15304 leader_election.cc:341] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80ebf772f86d4df396ff05b6b433e785 (127.0.98.2:37059): Not found: Tablet not found: 31f9ed0c9b11424fb0b3ac8d1d2253e1
I20260929 01:04:49.964433 15304 leader_election.cc:302] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aee98792440b4cad86a9757cd709a314; no voters: 50409a27acc24ff0b455aa410bfd076d, 80ebf772f86d4df396ff05b6b433e785
I20260929 01:04:49.964711 15690 raft_consensus.cc:2726] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:04:50.446643 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:50.446368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.1:47553)
0929 01:04:50.446501 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:50.446557 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 01:04:50.565299 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:50.564909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.2:52813)
0929 01:04:50.565153 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:50.565210 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":69,"threads_started":1}
I20260929 01:04:50.739902 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:50.739468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.3:58999)
0929 01:04:50.739750 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:50.739808 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":73,"threads_started":1}
I20260929 01:04:51.447707 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:51.447328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.1:35507)
0929 01:04:51.447558 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:51.447608 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":71,"threads_started":1}
I20260929 01:04:51.566418 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:51.566016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.2:46813)
0929 01:04:51.566264 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:51.566317 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":74,"threads_started":1}
I20260929 01:04:51.741030 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:51.740579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.3:55673)
0929 01:04:51.740875 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:51.740929 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":81,"threads_started":1}
I20260929 01:04:52.137125 15700 raft_consensus.cc:482] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:04:52.137250 15700 raft_consensus.cc:504] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } }
I20260929 01:04:52.137733 15700 leader_election.cc:288] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 80ebf772f86d4df396ff05b6b433e785 (127.0.98.2:37059), 50409a27acc24ff0b455aa410bfd076d (127.0.98.3:33801)
W20260929 01:04:52.138481 15303 leader_election.cc:341] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 50409a27acc24ff0b455aa410bfd076d (127.0.98.3:33801): Not found: Tablet not found: 31f9ed0c9b11424fb0b3ac8d1d2253e1
W20260929 01:04:52.138628 15304 leader_election.cc:341] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80ebf772f86d4df396ff05b6b433e785 (127.0.98.2:37059): Not found: Tablet not found: 31f9ed0c9b11424fb0b3ac8d1d2253e1
I20260929 01:04:52.138723 15304 leader_election.cc:302] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aee98792440b4cad86a9757cd709a314; no voters: 50409a27acc24ff0b455aa410bfd076d, 80ebf772f86d4df396ff05b6b433e785
I20260929 01:04:52.138901 15700 raft_consensus.cc:2726] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:04:52.448846 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:52.448457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.1:55429)
0929 01:04:52.448685 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:52.448734 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":73,"threads_started":1}
I20260929 01:04:52.567512 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:52.567131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.2:44171)
0929 01:04:52.567369 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:52.567425 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":69,"threads_started":1}
I20260929 01:04:52.742024 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:52.741626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.3:54623)
0929 01:04:52.741889 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:52.741941 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":103,"threads_started":1}
I20260929 01:04:53.450047 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:53.449635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.1:51087)
0929 01:04:53.449902 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:53.449955 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":69,"threads_started":1}
I20260929 01:04:53.568723 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:53.568233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.2:34851)
0929 01:04:53.568572 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:53.568629 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":100,"threads_started":1}
I20260929 01:04:53.743328 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:53.742787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.3:53777)
0929 01:04:53.743167 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:53.743235 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":70,"threads_started":1}
I20260929 01:04:54.448037 15707 raft_consensus.cc:482] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:04:54.448171 15707 raft_consensus.cc:504] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } }
I20260929 01:04:54.448666 15707 leader_election.cc:288] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 80ebf772f86d4df396ff05b6b433e785 (127.0.98.2:37059), 50409a27acc24ff0b455aa410bfd076d (127.0.98.3:33801)
W20260929 01:04:54.449534 15303 leader_election.cc:341] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 50409a27acc24ff0b455aa410bfd076d (127.0.98.3:33801): Not found: Tablet not found: 31f9ed0c9b11424fb0b3ac8d1d2253e1
W20260929 01:04:54.449630 15304 leader_election.cc:341] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80ebf772f86d4df396ff05b6b433e785 (127.0.98.2:37059): Not found: Tablet not found: 31f9ed0c9b11424fb0b3ac8d1d2253e1
I20260929 01:04:54.449673 15303 leader_election.cc:302] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aee98792440b4cad86a9757cd709a314; no voters: 50409a27acc24ff0b455aa410bfd076d, 80ebf772f86d4df396ff05b6b433e785
I20260929 01:04:54.449844 15707 raft_consensus.cc:2726] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:04:54.451074 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:54.450678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.1:42803)
0929 01:04:54.450944 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:54.450995 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":66,"threads_started":1}
I20260929 01:04:54.569968 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:54.569575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.2:48855)
0929 01:04:54.569825 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:54.569878 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":89,"threads_started":1}
I20260929 01:04:54.744376 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:54.743982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.3:58623)
0929 01:04:54.744240 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:54.744292 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":86,"threads_started":1}
I20260929 01:04:55.452234 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:55.451790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.1:47141)
0929 01:04:55.452071 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:55.452130 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":73,"threads_started":1}
I20260929 01:04:55.571138 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:55.570671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.2:50809)
0929 01:04:55.570991 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:55.571047 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":109,"threads_started":1}
I20260929 01:04:55.745548 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:55.745052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.3:41897)
0929 01:04:55.745398 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:55.745456 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":104,"threads_started":1}
I20260929 01:04:56.453358 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:56.452954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.1:60483)
0929 01:04:56.453205 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:56.453257 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":74,"threads_started":1}
I20260929 01:04:56.572227 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:56.571865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.2:36655)
0929 01:04:56.572088 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:56.572141 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":65,"threads_started":1}
I20260929 01:04:56.746682 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:56.746300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.3:34409)
0929 01:04:56.746539 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:56.746589 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":83,"threads_started":1}
I20260929 01:04:57.454478 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:57.454086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.1:59999)
0929 01:04:57.454325 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:57.454378 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260929 01:04:57.573422 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:57.572999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.2:35921)
0929 01:04:57.573271 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:57.573334 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":74,"threads_started":1}
I20260929 01:04:57.747886 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:57.747442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.3:38777)
0929 01:04:57.747709 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:57.747771 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":81,"threads_started":1}
I20260929 01:04:58.343813 15720 raft_consensus.cc:482] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:04:58.343942 15720 raft_consensus.cc:504] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } }
I20260929 01:04:58.344359 15720 leader_election.cc:288] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 80ebf772f86d4df396ff05b6b433e785 (127.0.98.2:37059), 50409a27acc24ff0b455aa410bfd076d (127.0.98.3:33801)
W20260929 01:04:58.345243 15303 leader_election.cc:341] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 50409a27acc24ff0b455aa410bfd076d (127.0.98.3:33801): Not found: Tablet not found: 31f9ed0c9b11424fb0b3ac8d1d2253e1
W20260929 01:04:58.345436 15304 leader_election.cc:341] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80ebf772f86d4df396ff05b6b433e785 (127.0.98.2:37059): Not found: Tablet not found: 31f9ed0c9b11424fb0b3ac8d1d2253e1
I20260929 01:04:58.345523 15304 leader_election.cc:302] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aee98792440b4cad86a9757cd709a314; no voters: 50409a27acc24ff0b455aa410bfd076d, 80ebf772f86d4df396ff05b6b433e785
I20260929 01:04:58.345713 15720 raft_consensus.cc:2726] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:04:58.455890 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:58.455429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.1:53525)
0929 01:04:58.455726 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:58.455793 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":81,"threads_started":1}
I20260929 01:04:58.574783 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:58.574277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.2:57867)
0929 01:04:58.574603 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:58.574676 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":74,"threads_started":1}
I20260929 01:04:58.749202 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:58.748770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.3:40209)
0929 01:04:58.749022 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:58.749089 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":72,"threads_started":1}
I20260929 01:04:59.457186 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:59.456749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.1:56315)
0929 01:04:59.457020 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:59.457084 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260929 01:04:59.576105 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:59.575542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.2:42571)
0929 01:04:59.575870 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:59.575962 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":96,"threads_started":1}
I20260929 01:04:59.750401 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:04:59.749965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.3:33571)
0929 01:04:59.750244 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 01:04:59.750303 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":113,"threads_started":1}
I20260929 01:05:00.458305 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:00.457901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.1:49357)
0929 01:05:00.458147 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:00.458198 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
I20260929 01:05:00.577302 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:00.576904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.2:54813)
0929 01:05:00.577161 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:00.577214 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":84,"threads_started":1}
I20260929 01:05:00.751603 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:00.751216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.3:35429)
0929 01:05:00.751455 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:00.751509 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":77,"threads_started":1}
I20260929 01:05:01.459609 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:01.459211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.1:49611)
0929 01:05:01.459453 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:01.459505 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":79,"threads_started":1}
I20260929 01:05:01.461064 15731 raft_consensus.cc:482] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:05:01.461170 15731 raft_consensus.cc:504] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ebf772f86d4df396ff05b6b433e785" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37059 } } peers { permanent_uuid: "50409a27acc24ff0b455aa410bfd076d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33801 } } peers { permanent_uuid: "aee98792440b4cad86a9757cd709a314" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33439 } }
I20260929 01:05:01.461611 15731 leader_election.cc:288] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 80ebf772f86d4df396ff05b6b433e785 (127.0.98.2:37059), 50409a27acc24ff0b455aa410bfd076d (127.0.98.3:33801)
W20260929 01:05:01.462301 15304 leader_election.cc:341] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 80ebf772f86d4df396ff05b6b433e785 (127.0.98.2:37059): Not found: Tablet not found: 31f9ed0c9b11424fb0b3ac8d1d2253e1
W20260929 01:05:01.462420 15303 leader_election.cc:341] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 50409a27acc24ff0b455aa410bfd076d (127.0.98.3:33801): Not found: Tablet not found: 31f9ed0c9b11424fb0b3ac8d1d2253e1
I20260929 01:05:01.462518 15303 leader_election.cc:302] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aee98792440b4cad86a9757cd709a314; no voters: 50409a27acc24ff0b455aa410bfd076d, 80ebf772f86d4df396ff05b6b433e785
I20260929 01:05:01.462713 15731 raft_consensus.cc:2726] T 31f9ed0c9b11424fb0b3ac8d1d2253e1 P aee98792440b4cad86a9757cd709a314 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:05:01.578462 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:01.578043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.2:39147)
0929 01:05:01.578321 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:01.578374 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":89,"threads_started":1}
I20260929 01:05:01.752806 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:01.752378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.3:42489)
0929 01:05:01.752661 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:01.752716 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":119,"threads_started":1}
I20260929 01:05:02.460817 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:02.460428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.1:45187)
0929 01:05:02.460671 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:02.460726 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":76,"threads_started":1}
I20260929 01:05:02.579766 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:02.579376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.2:54353)
0929 01:05:02.579620 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:02.579673 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":80,"threads_started":1}
I20260929 01:05:02.754112 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:02.753589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.3:59869)
0929 01:05:02.753907 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:02.753979 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
I20260929 01:05:03.462200 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:03.461802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.1:58201)
0929 01:05:03.462050 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:03.462107 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":81,"threads_started":1}
I20260929 01:05:03.581009 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:03.580610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.2:53245)
0929 01:05:03.580868 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:03.580920 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":95,"threads_started":1}
I20260929 01:05:03.743944 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 15253
I20260929 01:05:03.750047 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 15420
I20260929 01:05:03.755442 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:03.755044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.3:60189)
0929 01:05:03.755295 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:03.755346 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":82,"threads_started":1}
I20260929 01:05:03.755709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 15554
2026-09-29T01:05:03Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19399 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:05:03 dist-test-slave-7zdl krb5kdc[15742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:05:03 dist-test-slave-7zdl krb5kdc[15742](info): set up 2 sockets
Sep 29 01:05:03 dist-test-slave-7zdl krb5kdc[15742](info): commencing operation
krb5kdc: starting...
W20260929 01:05:04.794919 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:05:04 dist-test-slave-7zdl krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:05:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:05:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:05:04.915299 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42963
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:05:05.024228 15755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:05.024478 15755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:05.024680 15755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:05.028786 15755 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 01:05:05.028836 15755 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 01:05:05.028851 15755 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 01:05:05.028865 15755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:05:05.028883 15755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:05.028900 15755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:05.028918 15755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:05:05.028934 15755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:05:05.034137 15755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:05:05.035120 15755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:05.036062 15755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:05.044147 15763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:05.044203 15761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:05.044206 15760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:05.044306 15755 server_base.cc:1034] running on GCE node
I20260929 01:05:05.044914 15755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:05.045928 15755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:05.047087 15755 hybrid_clock.cc:648] HybridClock initialized: now 1790643905047062 us; error 39 us; skew 500 ppm
Sep 29 01:05:05 dist-test-slave-7zdl krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:05.056893 15755 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:05:05.057770 15755 webserver.cc:460] Webserver started at http://127.0.98.62:33669/ using document root <none> and password file <none>
I20260929 01:05:05.058328 15755 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:05.058418 15755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:05.058648 15755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:05.060668 15755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "beea0aa0698d42feb60125ad540762dd"
format_stamp: "Formatted at 2026-09-29 01:05:05 on dist-test-slave-7zdl"
server_key: "2af0b61038c6efa396007027520720c0"
server_key_iv: "6c53a402d7ed9197f12089b407214405"
server_key_version: "encryptionkey@0"
I20260929 01:05:05.061199 15755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "beea0aa0698d42feb60125ad540762dd"
format_stamp: "Formatted at 2026-09-29 01:05:05 on dist-test-slave-7zdl"
server_key: "2af0b61038c6efa396007027520720c0"
server_key_iv: "6c53a402d7ed9197f12089b407214405"
server_key_version: "encryptionkey@0"
I20260929 01:05:05.064759 15755 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:05:05.067072 15770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:05.068217 15755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:05:05.068361 15755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "beea0aa0698d42feb60125ad540762dd"
format_stamp: "Formatted at 2026-09-29 01:05:05 on dist-test-slave-7zdl"
server_key: "2af0b61038c6efa396007027520720c0"
server_key_iv: "6c53a402d7ed9197f12089b407214405"
server_key_version: "encryptionkey@0"
I20260929 01:05:05.068467 15755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:05.081413 15755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:05.084123 15755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:05.084313 15755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:05.091013 15755 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42963
I20260929 01:05:05.091295 15822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42963 every 8 connection(s)
I20260929 01:05:05.091943 15755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:05:05.094616 15823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:05.100350 15823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd: Bootstrap starting.
I20260929 01:05:05.100948 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 15755
I20260929 01:05:05.101056 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:05:05.101342 392 external_mini_cluster.cc:1428] Setting key 00da9c3a12ecc589bc2a5a0d782d0aea
I20260929 01:05:05.103226 15823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:05.103957 15823 log.cc:824] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:05.105998 15823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd: No bootstrap required, opened a new log
Sep 29 01:05:05 dist-test-slave-7zdl krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:05:05.110484 15823 raft_consensus.cc:348] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beea0aa0698d42feb60125ad540762dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42963 } }
I20260929 01:05:05.110707 15823 raft_consensus.cc:374] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:05.110764 15823 raft_consensus.cc:729] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beea0aa0698d42feb60125ad540762dd, State: Initialized, Role: FOLLOWER
I20260929 01:05:05.111326 15823 consensus_queue.cc:260] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beea0aa0698d42feb60125ad540762dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42963 } }
I20260929 01:05:05.111514 15823 raft_consensus.cc:388] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:05:05.111591 15823 raft_consensus.cc:482] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:05:05.111702 15823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:05.112890 15823 raft_consensus.cc:504] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beea0aa0698d42feb60125ad540762dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42963 } }
I20260929 01:05:05.113277 15823 leader_election.cc:302] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: beea0aa0698d42feb60125ad540762dd; no voters:
I20260929 01:05:05.113646 15823 leader_election.cc:288] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:05:05.113763 15828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:05:05.113999 15828 raft_consensus.cc:686] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [term 1 LEADER]: Becoming Leader. State: Replica: beea0aa0698d42feb60125ad540762dd, State: Running, Role: LEADER
I20260929 01:05:05.114368 15828 consensus_queue.cc:237] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beea0aa0698d42feb60125ad540762dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42963 } }
I20260929 01:05:05.115013 15823 sys_catalog.cc:564] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:05:05.116006 15830 sys_catalog.cc:455] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader beea0aa0698d42feb60125ad540762dd. Latest consensus state: current_term: 1 leader_uuid: "beea0aa0698d42feb60125ad540762dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beea0aa0698d42feb60125ad540762dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42963 } } }
I20260929 01:05:05.116128 15830 sys_catalog.cc:458] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [sys.catalog]: This master's current role is: LEADER
I20260929 01:05:05.116442 15829 sys_catalog.cc:455] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "beea0aa0698d42feb60125ad540762dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beea0aa0698d42feb60125ad540762dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42963 } } }
I20260929 01:05:05.116473 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:05.102619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55290 (local address 127.0.98.62:42963)
0929 01:05:05.103019 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:05.103035 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:05.103071 (+ 36us) server_negotiation.cc:407] Connection header received
0929 01:05:05.103771 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:05.103788 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:05.104157 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:05.104499 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:05.105277 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.106286 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.107418 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.107869 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.107982 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:05.110897 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:05.110904 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:05.110919 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:05.110949 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:05.113003 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:05.113506 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:05.113512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:05.113519 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:05.113677 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:05.113993 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:05.113997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:05.113999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:05.114485 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:05.114690 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:05.114985 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 01:05:05.115438 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":143,"threads_started":1}
I20260929 01:05:05.116518 15829 sys_catalog.cc:458] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [sys.catalog]: This master's current role is: LEADER
W20260929 01:05:05.119767 15843 catalog_manager.cc:1558] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:05:05.119858 15843 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:05:05.119930 15837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:05:05.120607 15837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:05:05.126112 15837 catalog_manager.cc:1347] Generated new cluster ID: d9496da45080486e93791d0f00a5982c
I20260929 01:05:05.126178 15837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:05:05.144547 15837 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:05:05.145457 15837 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:05:05.150839 15837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd: Generated new TSK 0
I20260929 01:05:05.151640 15837 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 01:05:05.181442 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42963
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:05:05.292622 15849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:05.292882 15849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:05.292925 15849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:05.296990 15849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:05.297047 15849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:05.297166 15849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:05:05.302343 15849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:05:05.303434 15849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:05.304461 15849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:05:05.313953 15849 server_base.cc:1034] running on GCE node
W20260929 01:05:05.313961 15854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:05.314009 15857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:05.313966 15855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:05.314525 15849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:05.315270 15849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:05.316424 15849 hybrid_clock.cc:648] HybridClock initialized: now 1790643905316394 us; error 45 us; skew 500 ppm
Sep 29 01:05:05 dist-test-slave-7zdl krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:05.319108 15849 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:05:05.320012 15849 webserver.cc:460] Webserver started at http://127.0.98.1:37291/ using document root <none> and password file <none>
I20260929 01:05:05.320568 15849 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:05.320636 15849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:05.320866 15849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:05.322912 15849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "0214c85c126c46599e8b3d8023c5ee7a"
format_stamp: "Formatted at 2026-09-29 01:05:05 on dist-test-slave-7zdl"
server_key: "c29be331a1fdad92a67717e48aa5565d"
server_key_iv: "a9b993c47c0c6d6a2f6d27e51607d6a8"
server_key_version: "encryptionkey@0"
I20260929 01:05:05.323483 15849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0214c85c126c46599e8b3d8023c5ee7a"
format_stamp: "Formatted at 2026-09-29 01:05:05 on dist-test-slave-7zdl"
server_key: "c29be331a1fdad92a67717e48aa5565d"
server_key_iv: "a9b993c47c0c6d6a2f6d27e51607d6a8"
server_key_version: "encryptionkey@0"
I20260929 01:05:05.327260 15849 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 01:05:05.329623 15864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:05.330933 15849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:05:05.331025 15849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "0214c85c126c46599e8b3d8023c5ee7a"
format_stamp: "Formatted at 2026-09-29 01:05:05 on dist-test-slave-7zdl"
server_key: "c29be331a1fdad92a67717e48aa5565d"
server_key_iv: "a9b993c47c0c6d6a2f6d27e51607d6a8"
server_key_version: "encryptionkey@0"
I20260929 01:05:05.331110 15849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:05.349400 15849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:05.352650 15849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:05.352797 15849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:05.353298 15849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:05.354050 15849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:05.354096 15849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:05.354135 15849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:05.354156 15849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:05.363225 15849 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40049
I20260929 01:05:05.363243 15977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40049 every 8 connection(s)
I20260929 01:05:05.364082 15849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:05:05.367568 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 15849
I20260929 01:05:05.367662 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:05:05.367957 392 external_mini_cluster.cc:1428] Setting key e8b1c91b8bd787b88c5d3dcea08f7c77
Sep 29 01:05:05 dist-test-slave-7zdl krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 01:05:05.378602 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:05.365730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45411 (local address 127.0.98.62:42963)
0929 01:05:05.365858 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:05.365862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:05.366598 (+ 736us) server_negotiation.cc:407] Connection header received
0929 01:05:05.367982 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:05.367985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:05.368044 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:05.368165 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:05.369447 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.369982 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.371546 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.371775 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.371908 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:05.375023 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:05.375025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:05.375027 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:05.375054 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:05.376443 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:05.377069 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:05.377071 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:05.377072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:05.377144 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:05.377548 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:05.377551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:05.377555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:05.377725 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:05.377798 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:05.378383 (+ 585us) server_negotiation.cc:299] Negotiation successful
0929 01:05:05.378481 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 01:05:05.379417 15980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:05.366035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.1:45411)
0929 01:05:05.366448 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:05.366497 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:05.367775 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:05.368335 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:05.368344 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:05.368753 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:05:05.369296 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:05.369316 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.370110 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.370114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:05.371426 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:05.371439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.371966 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.371973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:05.372196 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:05.372799 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:05:05.372823 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:05:05.374855 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:05:05.376567 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:05.376573 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:05.376587 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:05.376953 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:05.377240 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:05.377245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:05.377250 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:05.377440 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:05.377836 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:05:05.377845 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:05:05.378168 (+ 323us) client_negotiation.cc:769] Sending connection context
0929 01:05:05.378320 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 01:05:05.378556 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 01:05:05.380656 15978 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42963
I20260929 01:05:05.380901 15978 heartbeater.cc:464] Registering TS with master...
I20260929 01:05:05.381362 15978 heartbeater.cc:511] Master 127.0.98.62:42963 requested a full tablet report, sending...
I20260929 01:05:05.382916 15787 ts_manager.cc:194] Registered new tserver with Master: 0214c85c126c46599e8b3d8023c5ee7a (127.0.98.1:40049)
I20260929 01:05:05.384146 15787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45411
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 01:05:05.395771 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42963
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:05:05.504087 15983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:05.504338 15983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:05.504381 15983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:05.508404 15983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:05.508462 15983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:05.508569 15983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:05:05.513697 15983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:05:05.514726 15983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:05.515832 15983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:05.525357 15988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:05.525449 15983 server_base.cc:1034] running on GCE node
W20260929 01:05:05.525346 15989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:05.525346 15991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:05.526033 15983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:05.526628 15983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:05.527787 15983 hybrid_clock.cc:648] HybridClock initialized: now 1790643905527781 us; error 51 us; skew 500 ppm
Sep 29 01:05:05 dist-test-slave-7zdl krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:05.530428 15983 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:05:05.531406 15983 webserver.cc:460] Webserver started at http://127.0.98.2:33617/ using document root <none> and password file <none>
I20260929 01:05:05.531965 15983 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:05.532024 15983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:05.532269 15983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:05.534335 15983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "a3c65a34ee9d4056833bec226605e720"
format_stamp: "Formatted at 2026-09-29 01:05:05 on dist-test-slave-7zdl"
server_key: "ed5133ca58efdc874b66c594e7f6d1ad"
server_key_iv: "c5dc2b35e71c5a1635c517166b7cda71"
server_key_version: "encryptionkey@0"
I20260929 01:05:05.534952 15983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a3c65a34ee9d4056833bec226605e720"
format_stamp: "Formatted at 2026-09-29 01:05:05 on dist-test-slave-7zdl"
server_key: "ed5133ca58efdc874b66c594e7f6d1ad"
server_key_iv: "c5dc2b35e71c5a1635c517166b7cda71"
server_key_version: "encryptionkey@0"
I20260929 01:05:05.538407 15983 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:05:05.540742 15998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:05.541900 15983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 01:05:05.542032 15983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "a3c65a34ee9d4056833bec226605e720"
format_stamp: "Formatted at 2026-09-29 01:05:05 on dist-test-slave-7zdl"
server_key: "ed5133ca58efdc874b66c594e7f6d1ad"
server_key_iv: "c5dc2b35e71c5a1635c517166b7cda71"
server_key_version: "encryptionkey@0"
I20260929 01:05:05.542143 15983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:05.554908 15983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:05.557534 15983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:05.557739 15983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:05.558302 15983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:05.559109 15983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:05.559182 15983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:05.559263 15983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:05.559317 15983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:05.568056 15983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33785
I20260929 01:05:05.568100 16111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33785 every 8 connection(s)
I20260929 01:05:05.568925 15983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:05:05.571460 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 15983
I20260929 01:05:05.571609 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:05:05.571957 392 external_mini_cluster.cc:1428] Setting key c77b19e072c5f6ad614cefbecddcfb87
Sep 29 01:05:05 dist-test-slave-7zdl krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 01:05:05.582839 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:05.570632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41079 (local address 127.0.98.62:42963)
0929 01:05:05.570766 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:05.570769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:05.571695 (+ 926us) server_negotiation.cc:407] Connection header received
0929 01:05:05.573057 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:05.573060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:05.573115 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:05.573201 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:05.574301 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.575064 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.576166 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.576586 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.576719 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:05.579326 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:05.579329 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:05.579331 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:05.579354 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:05.580736 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:05.581316 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:05.581318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:05.581320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:05.581389 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:05.581794 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:05.581797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:05.581798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:05.581964 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:05.582038 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:05.582573 (+ 535us) server_negotiation.cc:299] Negotiation successful
0929 01:05:05.582675 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 01:05:05.583748 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:05.571059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.2:41079)
0929 01:05:05.571531 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:05.571581 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:05.572851 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:05.573358 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:05.573365 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:05.573710 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:05:05.574162 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:05.574175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.575190 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.575193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:05.576049 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:05.576059 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.576739 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.576742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:05.576879 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:05.577486 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:05:05.577502 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:05:05.579163 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:05:05.580856 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:05.580866 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:05.580881 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:05.581222 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:05.581483 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:05.581489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:05.581494 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:05.581691 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:05.582059 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:05:05.582064 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:05:05.582280 (+ 216us) client_negotiation.cc:769] Sending connection context
0929 01:05:05.582502 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 01:05:05.582757 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":132,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 01:05:05.584954 16112 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42963
I20260929 01:05:05.585209 16112 heartbeater.cc:464] Registering TS with master...
I20260929 01:05:05.585731 16112 heartbeater.cc:511] Master 127.0.98.62:42963 requested a full tablet report, sending...
I20260929 01:05:05.586778 15787 ts_manager.cc:194] Registered new tserver with Master: a3c65a34ee9d4056833bec226605e720 (127.0.98.2:33785)
I20260929 01:05:05.587378 15787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41079
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 01:05:05.597868 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42963
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:05:05.712206 16117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:05.712555 16117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:05.712630 16117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:05.717059 16117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:05.717123 16117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:05.717182 16117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:05:05.722653 16117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:05:05.723790 16117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:05.724830 16117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:05:05.734444 16117 server_base.cc:1034] running on GCE node
W20260929 01:05:05.734400 16123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:05.734433 16122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:05.734400 16125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:05.735114 16117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:05.735832 16117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:05.737015 16117 hybrid_clock.cc:648] HybridClock initialized: now 1790643905736991 us; error 35 us; skew 500 ppm
Sep 29 01:05:05 dist-test-slave-7zdl krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:05.739724 16117 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:05:05.740666 16117 webserver.cc:460] Webserver started at http://127.0.98.3:36393/ using document root <none> and password file <none>
I20260929 01:05:05.741246 16117 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:05.741339 16117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:05.741565 16117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:05:05.743702 16117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "676627090e2541f4b01d3612e2c0081d"
format_stamp: "Formatted at 2026-09-29 01:05:05 on dist-test-slave-7zdl"
server_key: "196bdf62a98d81ef4b64a11376058ea1"
server_key_iv: "dd92bd6c1340eae2bcc14d048a64bdf7"
server_key_version: "encryptionkey@0"
I20260929 01:05:05.744268 16117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "676627090e2541f4b01d3612e2c0081d"
format_stamp: "Formatted at 2026-09-29 01:05:05 on dist-test-slave-7zdl"
server_key: "196bdf62a98d81ef4b64a11376058ea1"
server_key_iv: "dd92bd6c1340eae2bcc14d048a64bdf7"
server_key_version: "encryptionkey@0"
I20260929 01:05:05.747975 16117 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:05:05.750415 16132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:05.751672 16117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:05:05.751811 16117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "676627090e2541f4b01d3612e2c0081d"
format_stamp: "Formatted at 2026-09-29 01:05:05 on dist-test-slave-7zdl"
server_key: "196bdf62a98d81ef4b64a11376058ea1"
server_key_iv: "dd92bd6c1340eae2bcc14d048a64bdf7"
server_key_version: "encryptionkey@0"
I20260929 01:05:05.751922 16117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:05.768188 16117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:05.770900 16117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:05.771075 16117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:05.771648 16117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:05.772641 16117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:05.772722 16117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:05.772785 16117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:05.772821 16117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:05.782473 16117 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36387
I20260929 01:05:05.782493 16245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36387 every 8 connection(s)
I20260929 01:05:05.783407 16117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:05:05.784364 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 16117
I20260929 01:05:05.784482 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:05:05.784752 392 external_mini_cluster.cc:1428] Setting key 3341f54883a7abc5614e8b395c2fa48b
Sep 29 01:05:05 dist-test-slave-7zdl krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:05:05.796661 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:05.785250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47361 (local address 127.0.98.62:42963)
0929 01:05:05.785398 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:05.785403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:05.786324 (+ 921us) server_negotiation.cc:407] Connection header received
0929 01:05:05.787703 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:05.787706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:05.787760 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:05.787833 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:05.788951 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.789486 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.790674 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.790979 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.791092 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:05.793387 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:05.793390 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:05.793393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:05.793418 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:05.794649 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:05.795167 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:05.795172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:05.795174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:05.795260 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:05.795591 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:05.795596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:05.795597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:05.795787 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:05.795893 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:05.796363 (+ 470us) server_negotiation.cc:299] Negotiation successful
0929 01:05:05.796481 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 01:05:05.797495 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:05.785732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.3:47361)
0929 01:05:05.786191 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:05.786233 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:05.787494 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:05.787998 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:05.788005 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:05.788355 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:05:05.788815 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:05.788828 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.789611 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.789614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:05.790497 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:05.790526 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.791146 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.791150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:05.791316 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:05.791824 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:05:05.791839 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:05:05.793229 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:05:05.794766 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:05.794771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:05.794785 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:05.795058 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:05.795357 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:05.795360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:05.795362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:05.795494 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:05.795890 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:05:05.795896 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:05:05.796138 (+ 242us) client_negotiation.cc:769] Sending connection context
0929 01:05:05.796324 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 01:05:05.796630 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":127,"threads_started":1}
I20260929 01:05:05.798712 16246 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42963
I20260929 01:05:05.798979 16246 heartbeater.cc:464] Registering TS with master...
I20260929 01:05:05.799451 16246 heartbeater.cc:511] Master 127.0.98.62:42963 requested a full tablet report, sending...
I20260929 01:05:05.800405 15787 ts_manager.cc:194] Registered new tserver with Master: 676627090e2541f4b01d3612e2c0081d (127.0.98.3:36387)
I20260929 01:05:05.800921 15787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47361
I20260929 01:05:05.810333 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:05:05.819267 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:05.812203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55304 (local address 127.0.98.62:42963)
0929 01:05:05.812339 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:05.812343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:05.812557 (+ 214us) server_negotiation.cc:407] Connection header received
0929 01:05:05.812618 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:05.812620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:05.812683 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:05.812755 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:05.813383 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.813917 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.814719 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.815011 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.815103 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:05.816460 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:05.816462 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:05.816464 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:05.816485 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:05.817859 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:05.818229 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:05.818231 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:05.818233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:05.818300 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:05.818512 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:05.818514 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:05.818515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:05.818718 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:05.818881 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:05.818991 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 01:05:05.819080 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 01:05:05.821940 15787 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:05:05.824898 15787 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:05:05.839109 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:05.833948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40049 (local address 127.0.0.1:41030)
0929 01:05:05.834210 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:05.834222 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:05.834307 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:05.835203 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:05.835206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:05.835226 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:05.835316 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:05.835322 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.836593 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.836596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:05.837732 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:05.837742 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.838576 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.838579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:05.838678 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:05.838711 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 01:05:05.838838 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 01:05:05.838952 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":42,"thread_start_us":129,"threads_started":1}
I20260929 01:05:05.839371 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:05.833198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36387 (local address 127.0.0.1:38802)
0929 01:05:05.833800 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:05.833849 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:05.833985 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:05.835381 (+ 1396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:05.835383 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:05.835401 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:05.835501 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:05.835507 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.836661 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.836665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:05.837883 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:05.837892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.838948 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.838952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:05.839045 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:05.839066 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 01:05:05.839139 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 01:05:05.839209 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":55,"spinlock_wait_cycles":8576,"thread_start_us":93,"threads_started":1}
I20260929 01:05:05.839445 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:05.833204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33730 (local address 127.0.98.2:33785)
0929 01:05:05.833525 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:05.833537 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:05.834116 (+ 579us) server_negotiation.cc:407] Connection header received
0929 01:05:05.834198 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:05.834203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:05.834409 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:05.834637 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:05.834915 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.836088 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.837739 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.838544 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.838662 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:05.838830 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:05.839182 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 01:05:05.839283 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":117,"threads_started":1}
I20260929 01:05:05.839742 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:05.833018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33785 (local address 127.0.0.1:33730)
0929 01:05:05.833800 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:05.833849 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:05.833977 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:05.834623 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:05.834628 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:05.834658 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:05.834757 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:05.834761 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.836224 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.836227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:05.837606 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:05.837616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.838786 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.838792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:05.838906 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:05.838926 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 01:05:05.838991 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 01:05:05.839062 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":85,"threads_started":1}
I20260929 01:05:05.839798 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:05.834307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41030 (local address 127.0.98.1:40049)
0929 01:05:05.834703 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:05.834716 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:05.834736 (+ 20us) server_negotiation.cc:407] Connection header received
0929 01:05:05.834835 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:05.834842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:05.835070 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:05.835214 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:05.835453 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.836442 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.837852 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.838446 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.838568 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:05.838711 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:05.839420 (+ 709us) server_negotiation.cc:299] Negotiation successful
0929 01:05:05.839654 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":90,"threads_started":1}
I20260929 01:05:05.839875 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:05.833682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38802 (local address 127.0.98.3:36387)
0929 01:05:05.834987 (+ 1305us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:05.834998 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:05.835018 (+ 20us) server_negotiation.cc:407] Connection header received
0929 01:05:05.835090 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:05.835095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:05.835268 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:05.835397 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:05.835634 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.836509 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.837997 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:05.838761 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:05.838950 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:05.839081 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:05.839178 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 01:05:05.839715 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1216,"thread_start_us":97,"threads_started":1}
I20260929 01:05:05.842381 16180 tablet_service.cc:1467] Processing CreateTablet for tablet d6ef6e042d4c4d779d8f0c1f004880ad (DEFAULT_TABLE table=test-table [id=baded28cb06e44b4870243290e014ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:05:05.842378 15912 tablet_service.cc:1467] Processing CreateTablet for tablet d6ef6e042d4c4d779d8f0c1f004880ad (DEFAULT_TABLE table=test-table [id=baded28cb06e44b4870243290e014ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:05:05.843339 15912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6ef6e042d4c4d779d8f0c1f004880ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:05.843348 16180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6ef6e042d4c4d779d8f0c1f004880ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:05.849830 16263 tablet_bootstrap.cc:492] T d6ef6e042d4c4d779d8f0c1f004880ad P 676627090e2541f4b01d3612e2c0081d: Bootstrap starting.
I20260929 01:05:05.849495 16046 tablet_service.cc:1467] Processing CreateTablet for tablet d6ef6e042d4c4d779d8f0c1f004880ad (DEFAULT_TABLE table=test-table [id=baded28cb06e44b4870243290e014ffa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:05:05.850564 16046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6ef6e042d4c4d779d8f0c1f004880ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:05.850983 16264 tablet_bootstrap.cc:492] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a: Bootstrap starting.
I20260929 01:05:05.851989 16263 tablet_bootstrap.cc:654] T d6ef6e042d4c4d779d8f0c1f004880ad P 676627090e2541f4b01d3612e2c0081d: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:05.853026 16263 log.cc:824] T d6ef6e042d4c4d779d8f0c1f004880ad P 676627090e2541f4b01d3612e2c0081d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:05.853029 16264 tablet_bootstrap.cc:654] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:05.854254 16264 log.cc:824] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:05.855041 16263 tablet_bootstrap.cc:492] T d6ef6e042d4c4d779d8f0c1f004880ad P 676627090e2541f4b01d3612e2c0081d: No bootstrap required, opened a new log
I20260929 01:05:05.855274 16263 ts_tablet_manager.cc:1397] T d6ef6e042d4c4d779d8f0c1f004880ad P 676627090e2541f4b01d3612e2c0081d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 01:05:05.856312 16264 tablet_bootstrap.cc:492] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a: No bootstrap required, opened a new log
I20260929 01:05:05.856498 16264 ts_tablet_manager.cc:1397] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 01:05:05.857430 16267 tablet_bootstrap.cc:492] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720: Bootstrap starting.
I20260929 01:05:05.859277 16263 raft_consensus.cc:348] T d6ef6e042d4c4d779d8f0c1f004880ad P 676627090e2541f4b01d3612e2c0081d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:05:05.859556 16263 raft_consensus.cc:374] T d6ef6e042d4c4d779d8f0c1f004880ad P 676627090e2541f4b01d3612e2c0081d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:05.859612 16263 raft_consensus.cc:729] T d6ef6e042d4c4d779d8f0c1f004880ad P 676627090e2541f4b01d3612e2c0081d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 676627090e2541f4b01d3612e2c0081d, State: Initialized, Role: FOLLOWER
I20260929 01:05:05.859694 16267 tablet_bootstrap.cc:654] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:05.860065 16263 consensus_queue.cc:260] T d6ef6e042d4c4d779d8f0c1f004880ad P 676627090e2541f4b01d3612e2c0081d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:05:05.860252 16264 raft_consensus.cc:348] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:05:05.860505 16264 raft_consensus.cc:374] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:05.860489 16267 log.cc:824] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:05.860560 16264 raft_consensus.cc:729] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0214c85c126c46599e8b3d8023c5ee7a, State: Initialized, Role: FOLLOWER
I20260929 01:05:05.860929 16246 heartbeater.cc:503] Master 127.0.98.62:42963 was elected leader, sending a full tablet report...
I20260929 01:05:05.861071 16263 ts_tablet_manager.cc:1428] T d6ef6e042d4c4d779d8f0c1f004880ad P 676627090e2541f4b01d3612e2c0081d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 01:05:05.860949 16264 consensus_queue.cc:260] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:05:05.862134 16264 ts_tablet_manager.cc:1428] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 01:05:05.862313 16267 tablet_bootstrap.cc:492] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720: No bootstrap required, opened a new log
I20260929 01:05:05.862464 16267 ts_tablet_manager.cc:1397] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 01:05:05.863008 15978 heartbeater.cc:503] Master 127.0.98.62:42963 was elected leader, sending a full tablet report...
W20260929 01:05:05.865134 15979 tablet.cc:2369] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:05:05.865957 16267 raft_consensus.cc:348] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:05:05.866147 16267 raft_consensus.cc:374] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:05.866179 16267 raft_consensus.cc:729] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3c65a34ee9d4056833bec226605e720, State: Initialized, Role: FOLLOWER
I20260929 01:05:05.866505 16267 consensus_queue.cc:260] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:05:05.867062 16112 heartbeater.cc:503] Master 127.0.98.62:42963 was elected leader, sending a full tablet report...
I20260929 01:05:05.867322 16267 ts_tablet_manager.cc:1428] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260929 01:05:06.034482 16247 tablet.cc:2369] T d6ef6e042d4c4d779d8f0c1f004880ad P 676627090e2541f4b01d3612e2c0081d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:05:06.070243 16113 tablet.cc:2369] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:05:06.082460 16271 raft_consensus.cc:482] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:05:06.082651 16271 raft_consensus.cc:504] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:05:06.083900 16271 leader_election.cc:288] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 676627090e2541f4b01d3612e2c0081d (127.0.98.3:36387), 0214c85c126c46599e8b3d8023c5ee7a (127.0.98.1:40049)
I20260929 01:05:06.088629 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:06.084268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46493 (local address 127.0.98.3:36387)
0929 01:05:06.084413 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:06.084419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:06.084729 (+ 310us) server_negotiation.cc:407] Connection header received
0929 01:05:06.084829 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:06.084831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:06.084873 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:06.084930 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:06.085191 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:06.085890 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:06.087380 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:06.087997 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:06.088105 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:06.088170 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:06.088429 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 01:05:06.088498 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 01:05:06.088634 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:06.084251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56383 (local address 127.0.98.1:40049)
0929 01:05:06.084367 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:06.084371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:06.084385 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:05:06.084549 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:06.084552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:06.084600 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:06.084676 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:06.085013 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:06.085811 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:06.087380 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:06.087997 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:06.088105 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:06.088171 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:06.088429 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 01:05:06.088498 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 01:05:06.088672 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:06.084143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40049 (local address 127.0.98.2:56383)
0929 01:05:06.084275 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:06.084291 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:06.084393 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:06.084730 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:06.084733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:06.084746 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:06.084882 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:06.084887 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:06.085914 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:06.085917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:06.087249 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:06.087260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:06.088124 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:06.088127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:06.088281 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:06.088323 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 01:05:06.088401 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 01:05:06.088510 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":55}
I20260929 01:05:06.088860 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:06.084143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36387 (local address 127.0.98.2:46493)
0929 01:05:06.084601 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:06.084612 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:06.084738 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:06.084973 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:06.084976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:06.084996 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:06.085090 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:06.085094 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:06.085987 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:06.085990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:06.087249 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:06.087260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:06.088124 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:06.088127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:06.088285 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:06.088330 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 01:05:06.088401 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 01:05:06.088510 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":212,"thread_start_us":117,"threads_started":1}
I20260929 01:05:06.089359 15932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ef6e042d4c4d779d8f0c1f004880ad" candidate_uuid: "a3c65a34ee9d4056833bec226605e720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0214c85c126c46599e8b3d8023c5ee7a" is_pre_election: true
I20260929 01:05:06.089461 16200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ef6e042d4c4d779d8f0c1f004880ad" candidate_uuid: "a3c65a34ee9d4056833bec226605e720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "676627090e2541f4b01d3612e2c0081d" is_pre_election: true
I20260929 01:05:06.089699 15932 raft_consensus.cc:2454] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3c65a34ee9d4056833bec226605e720 in term 0.
I20260929 01:05:06.089761 16200 raft_consensus.cc:2454] T d6ef6e042d4c4d779d8f0c1f004880ad P 676627090e2541f4b01d3612e2c0081d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3c65a34ee9d4056833bec226605e720 in term 0.
I20260929 01:05:06.090241 16002 leader_election.cc:302] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0214c85c126c46599e8b3d8023c5ee7a, 676627090e2541f4b01d3612e2c0081d, a3c65a34ee9d4056833bec226605e720; no voters:
I20260929 01:05:06.090519 16271 raft_consensus.cc:2781] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:05:06.090602 16271 raft_consensus.cc:482] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:05:06.090646 16271 raft_consensus.cc:3037] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:06.091686 16271 raft_consensus.cc:504] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:05:06.092036 16271 leader_election.cc:288] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720 [CANDIDATE]: Term 1 election: Requested vote from peers 676627090e2541f4b01d3612e2c0081d (127.0.98.3:36387), 0214c85c126c46599e8b3d8023c5ee7a (127.0.98.1:40049)
I20260929 01:05:06.092341 15932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ef6e042d4c4d779d8f0c1f004880ad" candidate_uuid: "a3c65a34ee9d4056833bec226605e720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0214c85c126c46599e8b3d8023c5ee7a"
I20260929 01:05:06.092336 16200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ef6e042d4c4d779d8f0c1f004880ad" candidate_uuid: "a3c65a34ee9d4056833bec226605e720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "676627090e2541f4b01d3612e2c0081d"
I20260929 01:05:06.092453 16200 raft_consensus.cc:3037] T d6ef6e042d4c4d779d8f0c1f004880ad P 676627090e2541f4b01d3612e2c0081d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:06.092453 15932 raft_consensus.cc:3037] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:06.093659 16200 raft_consensus.cc:2454] T d6ef6e042d4c4d779d8f0c1f004880ad P 676627090e2541f4b01d3612e2c0081d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3c65a34ee9d4056833bec226605e720 in term 1.
I20260929 01:05:06.093658 15932 raft_consensus.cc:2454] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3c65a34ee9d4056833bec226605e720 in term 1.
I20260929 01:05:06.093947 16000 leader_election.cc:302] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 676627090e2541f4b01d3612e2c0081d, a3c65a34ee9d4056833bec226605e720; no voters:
I20260929 01:05:06.094096 16271 raft_consensus.cc:2781] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:05:06.094327 16271 raft_consensus.cc:686] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720 [term 1 LEADER]: Becoming Leader. State: Replica: a3c65a34ee9d4056833bec226605e720, State: Running, Role: LEADER
I20260929 01:05:06.094650 16271 consensus_queue.cc:237] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:05:06.097759 15786 catalog_manager.cc:5495] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3c65a34ee9d4056833bec226605e720 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a3c65a34ee9d4056833bec226605e720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } health_report { overall_health: UNKNOWN } } }
I20260929 01:05:06.129027 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:06.125662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33742 (local address 127.0.98.2:33785)
0929 01:05:06.125786 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:06.125790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:06.125804 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:05:06.125938 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:06.125940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:06.125996 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:06.126100 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:05:06.126316 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:06.126826 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:06.127712 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:06.127913 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:06.127991 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:06.128238 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:05:06.128645 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:05:06.128731 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:06.128781 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 01:05:06.128826 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 01:05:06.134711 15932 raft_consensus.cc:1261] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Refusing update from remote peer a3c65a34ee9d4056833bec226605e720: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:05:06.134713 16200 raft_consensus.cc:1261] T d6ef6e042d4c4d779d8f0c1f004880ad P 676627090e2541f4b01d3612e2c0081d [term 1 FOLLOWER]: Refusing update from remote peer a3c65a34ee9d4056833bec226605e720: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:05:06.135381 16271 consensus_queue.cc:1035] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:05:06.135558 16273 consensus_queue.cc:1035] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:05:06.149644 15786 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:05:06.149813 15786 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:05:06.152196 15786 catalog_manager.cc:5782] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd: Sending DeleteTablet for 3 replicas of tablet d6ef6e042d4c4d779d8f0c1f004880ad
I20260929 01:05:06.152891 16046 tablet_service.cc:1514] Processing DeleteTablet for tablet d6ef6e042d4c4d779d8f0c1f004880ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:05:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33730
I20260929 01:05:06.152894 16180 tablet_service.cc:1514] Processing DeleteTablet for tablet d6ef6e042d4c4d779d8f0c1f004880ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:05:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38802
I20260929 01:05:06.153064 15912 tablet_service.cc:1514] Processing DeleteTablet for tablet d6ef6e042d4c4d779d8f0c1f004880ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:05:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41030
I20260929 01:05:06.153513 16287 tablet_replica.cc:330] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720: stopping tablet replica
I20260929 01:05:06.153524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 15849
I20260929 01:05:06.154196 16286 tablet_replica.cc:330] T d6ef6e042d4c4d779d8f0c1f004880ad P 676627090e2541f4b01d3612e2c0081d: stopping tablet replica
I20260929 01:05:06.153823 16287 raft_consensus.cc:2229] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:05:06.154589 16286 raft_consensus.cc:2229] T d6ef6e042d4c4d779d8f0c1f004880ad P 676627090e2541f4b01d3612e2c0081d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:05:06.154786 16287 raft_consensus.cc:2258] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:05:06.155000 16286 raft_consensus.cc:2258] T d6ef6e042d4c4d779d8f0c1f004880ad P 676627090e2541f4b01d3612e2c0081d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:05:06.156397 16286 ts_tablet_manager.cc:1905] T d6ef6e042d4c4d779d8f0c1f004880ad P 676627090e2541f4b01d3612e2c0081d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:05:06.156397 16287 ts_tablet_manager.cc:1905] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:05:06.159094 16287 ts_tablet_manager.cc:1918] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:05:06.159094 16286 ts_tablet_manager.cc:1918] T d6ef6e042d4c4d779d8f0c1f004880ad P 676627090e2541f4b01d3612e2c0081d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:05:06.159195 16287 log.cc:1196] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/wal/wals/d6ef6e042d4c4d779d8f0c1f004880ad
I20260929 01:05:06.159310 16286 log.cc:1196] T d6ef6e042d4c4d779d8f0c1f004880ad P 676627090e2541f4b01d3612e2c0081d: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/d6ef6e042d4c4d779d8f0c1f004880ad
I20260929 01:05:06.159543 16287 ts_tablet_manager.cc:1939] T d6ef6e042d4c4d779d8f0c1f004880ad P a3c65a34ee9d4056833bec226605e720: Deleting consensus metadata
I20260929 01:05:06.159746 16286 ts_tablet_manager.cc:1939] T d6ef6e042d4c4d779d8f0c1f004880ad P 676627090e2541f4b01d3612e2c0081d: Deleting consensus metadata
I20260929 01:05:06.160746 15771 catalog_manager.cc:4841] TS a3c65a34ee9d4056833bec226605e720 (127.0.98.2:33785): tablet d6ef6e042d4c4d779d8f0c1f004880ad (table test-table [id=baded28cb06e44b4870243290e014ffa]) successfully deleted
I20260929 01:05:06.160796 15772 catalog_manager.cc:4841] TS 676627090e2541f4b01d3612e2c0081d (127.0.98.3:36387): tablet d6ef6e042d4c4d779d8f0c1f004880ad (table test-table [id=baded28cb06e44b4870243290e014ffa]) successfully deleted
W20260929 01:05:06.163008 15774 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40049 (error 108)
I20260929 01:05:06.163285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 15983
I20260929 01:05:06.164837 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:06.164484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40049 (local address 127.0.0.1:41036)
0929 01:05:06.164664 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:06.164736 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260929 01:05:06.165010 15774 catalog_manager.cc:4584] TS 0214c85c126c46599e8b3d8023c5ee7a (127.0.98.1:40049): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d6ef6e042d4c4d779d8f0c1f004880ad: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40049: connect: Connection refused (error 111)
I20260929 01:05:06.169238 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 16117
I20260929 01:05:06.174681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 15755
I20260929 01:05:26.180869 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42963
--webserver_interface=127.0.98.62
--webserver_port=33669
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:05:26.290841 16290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:26.291119 16290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:26.291200 16290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:26.295267 16290 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 01:05:26.295325 16290 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 01:05:26.295341 16290 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 01:05:26.295356 16290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:05:26.295418 16290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:26.295459 16290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:26.295487 16290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:05:26.295516 16290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:05:26.302109 16290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:05:26.303284 16290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:26.304358 16290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:26.317939 16296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:26.318006 16299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:26.317914 16297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:26.318167 16290 server_base.cc:1034] running on GCE node
I20260929 01:05:26.318962 16290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:26.320240 16290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:26.321450 16290 hybrid_clock.cc:648] HybridClock initialized: now 1790643926321410 us; error 40 us; skew 500 ppm
Sep 29 01:05:26 dist-test-slave-7zdl krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:26.331955 16290 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:05:26.332981 16290 webserver.cc:460] Webserver started at http://127.0.98.62:33669/ using document root <none> and password file <none>
I20260929 01:05:26.333583 16290 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:26.333657 16290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:26.337792 16290 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:05:26.340081 16306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:26.341316 16290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:05:26.341467 16290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "beea0aa0698d42feb60125ad540762dd"
format_stamp: "Formatted at 2026-09-29 01:05:05 on dist-test-slave-7zdl"
server_key: "2af0b61038c6efa396007027520720c0"
server_key_iv: "6c53a402d7ed9197f12089b407214405"
server_key_version: "encryptionkey@0"
I20260929 01:05:26.342090 16290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:26.362033 16290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:26.365145 16290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:26.365306 16290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:26.372674 16290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42963
I20260929 01:05:26.372982 16358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42963 every 8 connection(s)
I20260929 01:05:26.373799 16290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:05:26.377714 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 16290
I20260929 01:05:26.378132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40049
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42963
--webserver_port=37291
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:05:26.378414 16359 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:05:26.380221 16359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd: Bootstrap starting.
I20260929 01:05:26.397351 16359 log.cc:824] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:26.405864 16359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:05:26.406244 16359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd: Bootstrap complete.
I20260929 01:05:26.410251 16359 raft_consensus.cc:348] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beea0aa0698d42feb60125ad540762dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42963 } }
I20260929 01:05:26.410724 16359 raft_consensus.cc:729] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: beea0aa0698d42feb60125ad540762dd, State: Initialized, Role: FOLLOWER
I20260929 01:05:26.411298 16359 consensus_queue.cc:260] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beea0aa0698d42feb60125ad540762dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42963 } }
I20260929 01:05:26.411445 16359 raft_consensus.cc:388] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:05:26.411526 16359 raft_consensus.cc:482] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:05:26.411643 16359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:05:26.413568 16359 raft_consensus.cc:504] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beea0aa0698d42feb60125ad540762dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42963 } }
I20260929 01:05:26.413902 16359 leader_election.cc:302] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: beea0aa0698d42feb60125ad540762dd; no voters:
I20260929 01:05:26.414215 16359 leader_election.cc:288] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 01:05:26.414356 16363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:05:26.414610 16363 raft_consensus.cc:686] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [term 2 LEADER]: Becoming Leader. State: Replica: beea0aa0698d42feb60125ad540762dd, State: Running, Role: LEADER
I20260929 01:05:26.415020 16363 consensus_queue.cc:237] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beea0aa0698d42feb60125ad540762dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42963 } }
I20260929 01:05:26.415537 16359 sys_catalog.cc:564] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:05:26.416591 16364 sys_catalog.cc:455] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "beea0aa0698d42feb60125ad540762dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beea0aa0698d42feb60125ad540762dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42963 } } }
I20260929 01:05:26.416635 16365 sys_catalog.cc:455] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader beea0aa0698d42feb60125ad540762dd. Latest consensus state: current_term: 2 leader_uuid: "beea0aa0698d42feb60125ad540762dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beea0aa0698d42feb60125ad540762dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42963 } } }
I20260929 01:05:26.416718 16364 sys_catalog.cc:458] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [sys.catalog]: This master's current role is: LEADER
I20260929 01:05:26.416749 16365 sys_catalog.cc:458] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [sys.catalog]: This master's current role is: LEADER
I20260929 01:05:26.417189 16372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:05:26.421610 16372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:05:26.422118 16372 catalog_manager.cc:1259] Loaded cluster ID: d9496da45080486e93791d0f00a5982c
I20260929 01:05:26.422207 16372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:05:26.424021 16372 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:05:26.424597 16372 catalog_manager.cc:5879] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd: Loaded TSK: 0
I20260929 01:05:26.435899 16372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd: Generated new TSK 1
I20260929 01:05:26.436172 16372 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 01:05:26.501959 16361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:26.502274 16361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:26.502352 16361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:26.506489 16361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:26.506577 16361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:26.506664 16361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:05:26.512200 16361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37291
--tserver_master_addrs=127.0.98.62:42963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:05:26.513227 16361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:26.514292 16361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:26.523954 16387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:26.523988 16389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:26.523955 16386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:26.524545 16361 server_base.cc:1034] running on GCE node
I20260929 01:05:26.525045 16361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:26.525843 16361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:26.527040 16361 hybrid_clock.cc:648] HybridClock initialized: now 1790643926527004 us; error 54 us; skew 500 ppm
Sep 29 01:05:26 dist-test-slave-7zdl krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:26.530311 16361 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:05:26.531329 16361 webserver.cc:460] Webserver started at http://127.0.98.1:37291/ using document root <none> and password file <none>
I20260929 01:05:26.531903 16361 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:26.532016 16361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:26.536186 16361 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:05:26.538518 16396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:26.539939 16361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:05:26.540076 16361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "0214c85c126c46599e8b3d8023c5ee7a"
format_stamp: "Formatted at 2026-09-29 01:05:05 on dist-test-slave-7zdl"
server_key: "c29be331a1fdad92a67717e48aa5565d"
server_key_iv: "a9b993c47c0c6d6a2f6d27e51607d6a8"
server_key_version: "encryptionkey@0"
I20260929 01:05:26.540642 16361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:26.564949 16361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:26.568439 16361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:26.568650 16361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:26.569320 16361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:26.570417 16403 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 01:05:26.574573 16361 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 01:05:26.574651 16361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 01:05:26.574726 16361 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 01:05:26.576473 16361 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 01:05:26.576543 16361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 01:05:26.576805 16403 tablet_bootstrap.cc:492] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a: Bootstrap starting.
I20260929 01:05:26.587194 16361 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40049
I20260929 01:05:26.587231 16510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40049 every 8 connection(s)
I20260929 01:05:26.588176 16361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:05:26.595191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 16361
I20260929 01:05:26.595634 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33785
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42963
--webserver_port=33617
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:05:26.596956 16403 log.cc:824] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:05:26 dist-test-slave-7zdl krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:05:26.605705 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:26.590517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43089 (local address 127.0.98.62:42963)
0929 01:05:26.591057 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:26.591077 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:26.591128 (+ 51us) server_negotiation.cc:407] Connection header received
0929 01:05:26.592301 (+ 1173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:26.592318 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:26.592764 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:26.593233 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:26.594286 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:26.595940 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:26.597549 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:26.598064 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:26.598238 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:26.600744 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:26.600748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:26.600762 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:26.600790 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:26.602490 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:26.603106 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:26.603112 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:26.603115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:26.603226 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:26.603566 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:26.603569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:26.603570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:26.603904 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:26.604040 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:26.604590 (+ 550us) server_negotiation.cc:299] Negotiation successful
0929 01:05:26.604847 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":139,"threads_started":1}
I20260929 01:05:26.605722 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:26.590459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.1:43089)
0929 01:05:26.590968 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:26.591024 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:26.591848 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:26.593240 (+ 1392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:26.593247 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:26.593634 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:05:26.594093 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:26.594106 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:26.596103 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:26.596108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:26.597422 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:26.597432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:26.598221 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:26.598224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:26.598362 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:26.599021 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:05:26.599037 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:05:26.600543 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:05:26.602625 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:26.602630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:26.602643 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:26.602978 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:26.603323 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:26.603327 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:26.603329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:26.603469 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:26.604025 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:05:26.604031 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:05:26.604283 (+ 252us) client_negotiation.cc:769] Sending connection context
0929 01:05:26.604457 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 01:05:26.604791 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":132,"threads_started":1}
I20260929 01:05:26.606945 16403 tablet_bootstrap.cc:492] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:05:26.607414 16403 tablet_bootstrap.cc:492] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a: Bootstrap complete.
I20260929 01:05:26.608281 16511 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42963
I20260929 01:05:26.608511 16511 heartbeater.cc:464] Registering TS with master...
I20260929 01:05:26.608908 16403 ts_tablet_manager.cc:1397] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a: Time spent bootstrapping tablet: real 0.032s user 0.025s sys 0.004s
I20260929 01:05:26.609012 16511 heartbeater.cc:511] Master 127.0.98.62:42963 requested a full tablet report, sending...
I20260929 01:05:26.611680 16323 ts_manager.cc:194] Registered new tserver with Master: 0214c85c126c46599e8b3d8023c5ee7a (127.0.98.1:40049)
I20260929 01:05:26.611860 16403 raft_consensus.cc:348] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:05:26.612370 16403 raft_consensus.cc:729] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0214c85c126c46599e8b3d8023c5ee7a, State: Initialized, Role: FOLLOWER
I20260929 01:05:26.612843 16403 consensus_queue.cc:260] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:05:26.613834 16403 ts_tablet_manager.cc:1428] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 01:05:26.615453 16323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43089
I20260929 01:05:26.618155 16511 heartbeater.cc:503] Master 127.0.98.62:42963 was elected leader, sending a full tablet report...
Sep 29 01:05:26 dist-test-slave-7zdl krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 01:05:26.625563 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:26.614765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40049 (local address 127.0.0.1:45236)
0929 01:05:26.615553 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:26.615595 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:26.615719 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:26.616374 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:26.616380 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:26.616834 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:05:26.616969 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:26.616978 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:26.618046 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:26.618051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:26.618921 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:26.618941 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:26.619806 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:26.619823 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:26.619940 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:26.620469 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:05:26.620492 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:05:26.621988 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:05:26.624007 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:26.624015 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:26.624020 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:26.624243 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:26.624543 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:26.624545 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:26.624546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:26.624619 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:26.625077 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:05:26.625082 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:05:26.625176 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 01:05:26.625285 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 01:05:26.625433 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":93,"threads_started":1}
I20260929 01:05:26.625713 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:26.614897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45236 (local address 127.0.98.1:40049)
0929 01:05:26.615326 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:26.615340 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:26.615681 (+ 341us) server_negotiation.cc:407] Connection header received
0929 01:05:26.615892 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:26.615897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:26.616116 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:26.616252 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:26.617196 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:26.617928 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:26.619052 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:26.619588 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:26.619696 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:26.622119 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:26.622128 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:26.622141 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:26.622165 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:26.623903 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:26.624346 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:26.624352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:26.624354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:26.624447 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:26.624698 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:26.624700 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:26.624702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:26.624977 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:26.625105 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:26.625382 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 01:05:26.625554 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":114,"threads_started":1}
I20260929 01:05:26.627053 16445 tablet_service.cc:1514] Processing DeleteTablet for tablet d6ef6e042d4c4d779d8f0c1f004880ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:05:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45236
I20260929 01:05:26.627905 16520 tablet_replica.cc:330] stopping tablet replica
I20260929 01:05:26.628201 16520 raft_consensus.cc:2229] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:05:26.628444 16520 raft_consensus.cc:2258] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:05:26.629390 16520 ts_tablet_manager.cc:1905] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:05:26.632966 16520 ts_tablet_manager.cc:1918] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:05:26.633047 16520 log.cc:1196] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/wal/wals/d6ef6e042d4c4d779d8f0c1f004880ad
I20260929 01:05:26.633298 16520 ts_tablet_manager.cc:1939] T d6ef6e042d4c4d779d8f0c1f004880ad P 0214c85c126c46599e8b3d8023c5ee7a: Deleting consensus metadata
I20260929 01:05:26.635015 16310 catalog_manager.cc:4841] TS 0214c85c126c46599e8b3d8023c5ee7a (127.0.98.1:40049): tablet d6ef6e042d4c4d779d8f0c1f004880ad (table test-table [id=baded28cb06e44b4870243290e014ffa]) successfully deleted
W20260929 01:05:26.720099 16515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:26.720427 16515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:26.720506 16515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:26.725109 16515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:26.725205 16515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:26.725306 16515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:05:26.730707 16515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33617
--tserver_master_addrs=127.0.98.62:42963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:05:26.731855 16515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:26.732919 16515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:26.742586 16528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:26.742584 16525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:26.742584 16526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:26.742646 16515 server_base.cc:1034] running on GCE node
I20260929 01:05:26.743268 16515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:26.743877 16515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:26.745075 16515 hybrid_clock.cc:648] HybridClock initialized: now 1790643926745056 us; error 32 us; skew 500 ppm
Sep 29 01:05:26 dist-test-slave-7zdl krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:26.747821 16515 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:05:26.748745 16515 webserver.cc:460] Webserver started at http://127.0.98.2:33617/ using document root <none> and password file <none>
I20260929 01:05:26.749301 16515 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:26.749402 16515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:26.753249 16515 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 01:05:26.755345 16535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:26.756526 16515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:05:26.756650 16515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "a3c65a34ee9d4056833bec226605e720"
format_stamp: "Formatted at 2026-09-29 01:05:05 on dist-test-slave-7zdl"
server_key: "ed5133ca58efdc874b66c594e7f6d1ad"
server_key_iv: "c5dc2b35e71c5a1635c517166b7cda71"
server_key_version: "encryptionkey@0"
I20260929 01:05:26.757169 16515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:26.781064 16515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:26.784281 16515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:26.784493 16515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:26.785105 16515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:26.785939 16515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:26.786006 16515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:26.786079 16515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:26.786118 16515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:26.794931 16515 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33785
I20260929 01:05:26.795006 16648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33785 every 8 connection(s)
I20260929 01:05:26.795953 16515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:05:26.803169 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 16515
I20260929 01:05:26.803602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36387
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42963
--webserver_port=36393
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:05:26 dist-test-slave-7zdl krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:05:26.809556 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:26.797785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47975 (local address 127.0.98.62:42963)
0929 01:05:26.797926 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:26.797930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:26.799288 (+ 1358us) server_negotiation.cc:407] Connection header received
0929 01:05:26.800195 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:26.800199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:26.800260 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:26.800364 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:26.801538 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:26.802074 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:26.803266 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:26.803511 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:26.803618 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:26.806018 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:26.806021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:26.806024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:26.806048 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:26.807519 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:26.808023 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:26.808027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:26.808029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:26.808110 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:26.808450 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:26.808453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:26.808455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:26.808634 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:26.808741 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:26.809234 (+ 493us) server_negotiation.cc:299] Negotiation successful
0929 01:05:26.809371 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 01:05:26.810451 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:26.798186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.2:47975)
0929 01:05:26.799134 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:26.799176 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:26.799988 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:26.800548 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:26.800555 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:26.800913 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:05:26.801388 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:26.801401 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:26.802200 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:26.802203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:26.803139 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:26.803149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:26.803670 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:26.803674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:26.803842 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:26.804444 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:05:26.804460 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:05:26.805843 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:05:26.807637 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:26.807643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:26.807659 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:26.807913 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:26.808198 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:26.808202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:26.808204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:26.808343 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:26.808738 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:05:26.808744 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:05:26.809005 (+ 261us) client_negotiation.cc:769] Sending connection context
0929 01:05:26.809196 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 01:05:26.809523 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":600,"threads_started":1}
I20260929 01:05:26.811769 16649 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42963
I20260929 01:05:26.812044 16649 heartbeater.cc:464] Registering TS with master...
I20260929 01:05:26.812533 16649 heartbeater.cc:511] Master 127.0.98.62:42963 requested a full tablet report, sending...
I20260929 01:05:26.813566 16323 ts_manager.cc:194] Registered new tserver with Master: a3c65a34ee9d4056833bec226605e720 (127.0.98.2:33785)
I20260929 01:05:26.814128 16323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47975
W20260929 01:05:26.915184 16652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:26.915480 16652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:26.915555 16652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:26.919927 16652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:26.919994 16652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:26.920056 16652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:05:26.925344 16652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36393
--tserver_master_addrs=127.0.98.62:42963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:05:26.926376 16652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:26.927434 16652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:26.937418 16660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:26.937453 16657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:26.937418 16658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:26.937809 16652 server_base.cc:1034] running on GCE node
I20260929 01:05:26.938194 16652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:26.938995 16652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:26.940173 16652 hybrid_clock.cc:648] HybridClock initialized: now 1790643926940145 us; error 49 us; skew 500 ppm
Sep 29 01:05:26 dist-test-slave-7zdl krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:26.942965 16652 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:05:26.943853 16652 webserver.cc:460] Webserver started at http://127.0.98.3:36393/ using document root <none> and password file <none>
I20260929 01:05:26.944394 16652 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:26.944521 16652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:26.948480 16652 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 01:05:26.951027 16667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:26.952324 16652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:05:26.952466 16652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "676627090e2541f4b01d3612e2c0081d"
format_stamp: "Formatted at 2026-09-29 01:05:05 on dist-test-slave-7zdl"
server_key: "196bdf62a98d81ef4b64a11376058ea1"
server_key_iv: "dd92bd6c1340eae2bcc14d048a64bdf7"
server_key_version: "encryptionkey@0"
I20260929 01:05:26.953002 16652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:26.980813 16652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:26.984107 16652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:26.984303 16652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:26.984968 16652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:26.985821 16652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:26.985873 16652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:26.985939 16652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:26.985972 16652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:26.994715 16652 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36387
I20260929 01:05:26.994742 16780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36387 every 8 connection(s)
I20260929 01:05:26.995767 16652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:05:26.999956 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 16652
Sep 29 01:05:27 dist-test-slave-7zdl krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:05:27.009564 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:27.000645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53694 (local address 127.0.98.62:42963)
0929 01:05:27.000893 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:27.000897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:27.001039 (+ 142us) server_negotiation.cc:407] Connection header received
0929 01:05:27.001156 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:27.001158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:27.001192 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:27.001265 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:27.002114 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.002761 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.003970 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.004277 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.004376 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:27.006065 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:27.006069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:27.006073 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:27.006104 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:27.007881 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:27.008320 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:27.008326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:27.008328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:27.008417 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:27.008693 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:27.008697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:27.008699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:27.008905 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:27.009119 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:27.009229 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 01:05:27.009357 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
I20260929 01:05:27.011089 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:26.997724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46295 (local address 127.0.98.62:42963)
0929 01:05:26.997896 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:26.997900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:26.998707 (+ 807us) server_negotiation.cc:407] Connection header received
0929 01:05:27.000065 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:27.000070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:27.000129 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:27.000227 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:27.001786 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.002487 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.004096 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.004399 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.004495 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:27.007391 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:27.007395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:27.007397 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:27.007464 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:27.008819 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:27.009399 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:27.009403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:27.009404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:27.009488 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:27.009874 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:27.009877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:27.009879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:27.010086 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:27.010159 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:27.010783 (+ 624us) server_negotiation.cc:299] Negotiation successful
0929 01:05:27.010949 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 01:05:27.011988 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:26.998101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.3:46295)
0929 01:05:26.998551 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:26.998592 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:26.999845 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:27.000391 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:27.000398 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:27.000907 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:05:27.001628 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:27.001649 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.002610 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.002612 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:27.004002 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:27.004017 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.004561 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.004566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:27.004776 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:27.005241 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:05:27.005264 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:05:27.007212 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:05:27.008923 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:27.008928 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:27.008944 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:27.009272 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:27.009571 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:27.009577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:27.009582 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:27.009764 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:27.010170 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:05:27.010176 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:05:27.010508 (+ 332us) client_negotiation.cc:769] Sending connection context
0929 01:05:27.010701 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 01:05:27.011057 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":99,"threads_started":1}
I20260929 01:05:27.013033 16781 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42963
I20260929 01:05:27.013449 16781 heartbeater.cc:464] Registering TS with master...
I20260929 01:05:27.013949 16781 heartbeater.cc:511] Master 127.0.98.62:42963 requested a full tablet report, sending...
I20260929 01:05:27.014936 16323 ts_manager.cc:194] Registered new tserver with Master: 676627090e2541f4b01d3612e2c0081d (127.0.98.3:36387)
I20260929 01:05:27.015463 16323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46295
I20260929 01:05:27.023384 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:05:27.027351 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:27.024036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53710 (local address 127.0.98.62:42963)
0929 01:05:27.024157 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:27.024160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:27.024387 (+ 227us) server_negotiation.cc:407] Connection header received
0929 01:05:27.024539 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:27.024541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:27.024636 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:27.024717 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:05:27.024969 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.025464 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.026274 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.026487 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.026559 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:27.026909 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:05:27.027046 (+ 137us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 01:05:27.027251 (+ 205us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53710: token expired
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 01:05:27.027456 16514 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53710: token expired
W20260929 01:05:27.027829 16251 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:42963)
I20260929 01:05:27.034482 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:27.028165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53712 (local address 127.0.98.62:42963)
0929 01:05:27.028266 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:27.028269 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:27.028280 (+ 11us) server_negotiation.cc:407] Connection header received
0929 01:05:27.028366 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:27.028368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:27.028402 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:27.028472 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:27.029058 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.029511 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.030239 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.030425 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.030495 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:27.031908 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:27.031911 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:27.031913 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:27.031941 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:27.033164 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:27.033559 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:27.033561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:27.033563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:27.033630 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:27.033835 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:27.033837 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:27.033838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:27.033999 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:27.034095 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:27.034288 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 01:05:27.034373 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 01:05:27.037014 16323 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:05:27.040275 16323 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:05:27.048242 16445 tablet_service.cc:1467] Processing CreateTablet for tablet eb55d4f12821418aa246761699f4cc22 (DEFAULT_TABLE table=test-table [id=1893a630e02542f09fa8123855f335e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:05:27.048894 16445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb55d4f12821418aa246761699f4cc22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:27.051611 16403 tablet_bootstrap.cc:492] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a: Bootstrap starting.
I20260929 01:05:27.052531 16403 tablet_bootstrap.cc:654] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:27.052771 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:27.047088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36387 (local address 127.0.0.1:43848)
0929 01:05:27.047210 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:27.047225 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:27.047302 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:27.048326 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:27.048329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:27.048347 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:27.048453 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:27.048459 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.050174 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.050177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:27.051287 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:27.051297 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.052349 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.052353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:27.052444 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:27.052484 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 01:05:27.052553 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 01:05:27.052638 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":127}
I20260929 01:05:27.052836 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:27.047294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43848 (local address 127.0.98.3:36387)
0929 01:05:27.047739 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:27.047751 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:27.047775 (+ 24us) server_negotiation.cc:407] Connection header received
0929 01:05:27.047867 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:27.047872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:27.048156 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:27.048762 (+ 606us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:27.048954 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.050044 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.051411 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.052215 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.052348 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:27.052512 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:27.052582 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 01:05:27.052678 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":126,"threads_started":1}
I20260929 01:05:27.053328 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:27.047377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33785 (local address 127.0.0.1:34456)
0929 01:05:27.047773 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:27.047785 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:27.047861 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:27.048608 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:27.048611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:27.048631 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:27.048724 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:27.048730 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.050145 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.050149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:27.051388 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:27.051398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.052953 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.052957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:27.053038 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:27.053061 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 01:05:27.053130 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 01:05:27.053200 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":3,"thread_start_us":115,"threads_started":1}
I20260929 01:05:27.053699 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:27.047803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34456 (local address 127.0.98.2:33785)
0929 01:05:27.048139 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:27.048155 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:27.048177 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:05:27.048253 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:27.048259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:27.048459 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:27.048618 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:27.048815 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.050021 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.051551 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.052219 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.052346 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:27.052510 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:27.053173 (+ 663us) server_negotiation.cc:299] Negotiation successful
0929 01:05:27.053268 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":125,"threads_started":1}
I20260929 01:05:27.054047 16403 tablet_bootstrap.cc:492] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a: No bootstrap required, opened a new log
I20260929 01:05:27.054149 16403 ts_tablet_manager.cc:1397] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 01:05:27.054621 16403 raft_consensus.cc:348] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:05:27.054746 16403 raft_consensus.cc:374] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:27.054783 16403 raft_consensus.cc:729] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0214c85c126c46599e8b3d8023c5ee7a, State: Initialized, Role: FOLLOWER
I20260929 01:05:27.055112 16403 consensus_queue.cc:260] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:05:27.055379 16403 ts_tablet_manager.cc:1428] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 01:05:27.056721 16583 tablet_service.cc:1467] Processing CreateTablet for tablet eb55d4f12821418aa246761699f4cc22 (DEFAULT_TABLE table=test-table [id=1893a630e02542f09fa8123855f335e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:05:27.056901 16715 tablet_service.cc:1467] Processing CreateTablet for tablet eb55d4f12821418aa246761699f4cc22 (DEFAULT_TABLE table=test-table [id=1893a630e02542f09fa8123855f335e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:05:27.057844 16583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb55d4f12821418aa246761699f4cc22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:27.057916 16715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb55d4f12821418aa246761699f4cc22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:05:27.064419 16795 tablet_bootstrap.cc:492] T eb55d4f12821418aa246761699f4cc22 P 676627090e2541f4b01d3612e2c0081d: Bootstrap starting.
I20260929 01:05:27.064407 16794 tablet_bootstrap.cc:492] T eb55d4f12821418aa246761699f4cc22 P a3c65a34ee9d4056833bec226605e720: Bootstrap starting.
I20260929 01:05:27.066601 16795 tablet_bootstrap.cc:654] T eb55d4f12821418aa246761699f4cc22 P 676627090e2541f4b01d3612e2c0081d: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:27.066752 16794 tablet_bootstrap.cc:654] T eb55d4f12821418aa246761699f4cc22 P a3c65a34ee9d4056833bec226605e720: Neither blocks nor log segments found. Creating new log.
I20260929 01:05:27.067479 16794 log.cc:824] T eb55d4f12821418aa246761699f4cc22 P a3c65a34ee9d4056833bec226605e720: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:27.067622 16795 log.cc:824] T eb55d4f12821418aa246761699f4cc22 P 676627090e2541f4b01d3612e2c0081d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:27.069157 16794 tablet_bootstrap.cc:492] T eb55d4f12821418aa246761699f4cc22 P a3c65a34ee9d4056833bec226605e720: No bootstrap required, opened a new log
I20260929 01:05:27.069165 16795 tablet_bootstrap.cc:492] T eb55d4f12821418aa246761699f4cc22 P 676627090e2541f4b01d3612e2c0081d: No bootstrap required, opened a new log
I20260929 01:05:27.069355 16794 ts_tablet_manager.cc:1397] T eb55d4f12821418aa246761699f4cc22 P a3c65a34ee9d4056833bec226605e720: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 01:05:27.069360 16795 ts_tablet_manager.cc:1397] T eb55d4f12821418aa246761699f4cc22 P 676627090e2541f4b01d3612e2c0081d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 01:05:27.072295 16795 raft_consensus.cc:348] T eb55d4f12821418aa246761699f4cc22 P 676627090e2541f4b01d3612e2c0081d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:05:27.072295 16794 raft_consensus.cc:348] T eb55d4f12821418aa246761699f4cc22 P a3c65a34ee9d4056833bec226605e720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:05:27.072512 16795 raft_consensus.cc:374] T eb55d4f12821418aa246761699f4cc22 P 676627090e2541f4b01d3612e2c0081d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:27.072538 16794 raft_consensus.cc:374] T eb55d4f12821418aa246761699f4cc22 P a3c65a34ee9d4056833bec226605e720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:05:27.072567 16795 raft_consensus.cc:729] T eb55d4f12821418aa246761699f4cc22 P 676627090e2541f4b01d3612e2c0081d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 676627090e2541f4b01d3612e2c0081d, State: Initialized, Role: FOLLOWER
I20260929 01:05:27.072602 16794 raft_consensus.cc:729] T eb55d4f12821418aa246761699f4cc22 P a3c65a34ee9d4056833bec226605e720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3c65a34ee9d4056833bec226605e720, State: Initialized, Role: FOLLOWER
I20260929 01:05:27.072996 16795 consensus_queue.cc:260] T eb55d4f12821418aa246761699f4cc22 P 676627090e2541f4b01d3612e2c0081d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:05:27.072996 16794 consensus_queue.cc:260] T eb55d4f12821418aa246761699f4cc22 P a3c65a34ee9d4056833bec226605e720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:05:27.073582 16781 heartbeater.cc:503] Master 127.0.98.62:42963 was elected leader, sending a full tablet report...
I20260929 01:05:27.073916 16795 ts_tablet_manager.cc:1428] T eb55d4f12821418aa246761699f4cc22 P 676627090e2541f4b01d3612e2c0081d: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260929 01:05:27.073917 16794 ts_tablet_manager.cc:1428] T eb55d4f12821418aa246761699f4cc22 P a3c65a34ee9d4056833bec226605e720: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260929 01:05:27.074136 16649 heartbeater.cc:503] Master 127.0.98.62:42963 was elected leader, sending a full tablet report...
I20260929 01:05:27.078277 16517 raft_consensus.cc:482] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:05:27.078481 16517 raft_consensus.cc:504] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:05:27.079528 16517 leader_election.cc:288] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 676627090e2541f4b01d3612e2c0081d (127.0.98.3:36387), a3c65a34ee9d4056833bec226605e720 (127.0.98.2:33785)
I20260929 01:05:27.083261 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:27.079679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36387 (local address 127.0.98.1:42649)
0929 01:05:27.079846 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:27.079858 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:27.079973 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:27.080342 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:27.080344 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:27.080358 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:27.080465 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:27.080471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.081169 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.081174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:27.082104 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:27.082120 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.082777 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.082780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:27.082899 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:27.082949 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 01:05:27.083057 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 01:05:27.083134 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":34}
I20260929 01:05:27.083261 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:27.079730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42649 (local address 127.0.98.3:36387)
0929 01:05:27.079854 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:27.079858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:27.079896 (+ 38us) server_negotiation.cc:407] Connection header received
0929 01:05:27.080102 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:27.080107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:27.080156 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:27.080234 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:27.080571 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.081068 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.082225 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.082665 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.082775 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:27.082865 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:27.083073 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 01:05:27.083139 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 01:05:27.083539 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:27.080150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46961 (local address 127.0.98.2:33785)
0929 01:05:27.080274 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:27.080278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:27.080294 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:05:27.080327 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:27.080328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:27.080360 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:27.080444 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:05:27.080697 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.081176 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.082343 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.082741 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.082877 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:27.082940 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:27.083343 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 01:05:27.083430 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 01:05:27.083930 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:27.079787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33785 (local address 127.0.98.1:46961)
0929 01:05:27.080073 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:27.080087 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:27.080155 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:27.080452 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:27.080455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:27.080471 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:05:27.080592 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:27.080604 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.081275 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.081280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:27.082235 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:27.082243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.082883 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.082887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:27.083001 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:27.083023 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 01:05:27.083706 (+ 683us) client_negotiation.cc:240] Negotiation successful
0929 01:05:27.083779 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":17,"thread_start_us":96,"threads_started":1}
I20260929 01:05:27.084100 16735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb55d4f12821418aa246761699f4cc22" candidate_uuid: "0214c85c126c46599e8b3d8023c5ee7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "676627090e2541f4b01d3612e2c0081d" is_pre_election: true
I20260929 01:05:27.084383 16735 raft_consensus.cc:2454] T eb55d4f12821418aa246761699f4cc22 P 676627090e2541f4b01d3612e2c0081d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0214c85c126c46599e8b3d8023c5ee7a in term 0.
I20260929 01:05:27.084448 16603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb55d4f12821418aa246761699f4cc22" candidate_uuid: "0214c85c126c46599e8b3d8023c5ee7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c65a34ee9d4056833bec226605e720" is_pre_election: true
I20260929 01:05:27.084683 16603 raft_consensus.cc:2454] T eb55d4f12821418aa246761699f4cc22 P a3c65a34ee9d4056833bec226605e720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0214c85c126c46599e8b3d8023c5ee7a in term 0.
I20260929 01:05:27.084867 16398 leader_election.cc:302] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0214c85c126c46599e8b3d8023c5ee7a, 676627090e2541f4b01d3612e2c0081d; no voters:
I20260929 01:05:27.085143 16517 raft_consensus.cc:2781] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:05:27.085217 16517 raft_consensus.cc:482] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:05:27.085268 16517 raft_consensus.cc:3037] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:27.086314 16517 raft_consensus.cc:504] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:05:27.086665 16517 leader_election.cc:288] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [CANDIDATE]: Term 1 election: Requested vote from peers 676627090e2541f4b01d3612e2c0081d (127.0.98.3:36387), a3c65a34ee9d4056833bec226605e720 (127.0.98.2:33785)
I20260929 01:05:27.086985 16735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb55d4f12821418aa246761699f4cc22" candidate_uuid: "0214c85c126c46599e8b3d8023c5ee7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "676627090e2541f4b01d3612e2c0081d"
I20260929 01:05:27.087015 16603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb55d4f12821418aa246761699f4cc22" candidate_uuid: "0214c85c126c46599e8b3d8023c5ee7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c65a34ee9d4056833bec226605e720"
I20260929 01:05:27.087114 16735 raft_consensus.cc:3037] T eb55d4f12821418aa246761699f4cc22 P 676627090e2541f4b01d3612e2c0081d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:27.087124 16603 raft_consensus.cc:3037] T eb55d4f12821418aa246761699f4cc22 P a3c65a34ee9d4056833bec226605e720 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:05:27.088078 16603 raft_consensus.cc:2454] T eb55d4f12821418aa246761699f4cc22 P a3c65a34ee9d4056833bec226605e720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0214c85c126c46599e8b3d8023c5ee7a in term 1.
I20260929 01:05:27.088102 16735 raft_consensus.cc:2454] T eb55d4f12821418aa246761699f4cc22 P 676627090e2541f4b01d3612e2c0081d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0214c85c126c46599e8b3d8023c5ee7a in term 1.
I20260929 01:05:27.088357 16397 leader_election.cc:302] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0214c85c126c46599e8b3d8023c5ee7a, a3c65a34ee9d4056833bec226605e720; no voters:
I20260929 01:05:27.088516 16517 raft_consensus.cc:2781] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:05:27.088749 16517 raft_consensus.cc:686] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 1 LEADER]: Becoming Leader. State: Replica: 0214c85c126c46599e8b3d8023c5ee7a, State: Running, Role: LEADER
I20260929 01:05:27.089005 16517 consensus_queue.cc:237] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
W20260929 01:05:27.089303 16512 tablet.cc:2369] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:05:27.092264 16321 catalog_manager.cc:5495] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 0214c85c126c46599e8b3d8023c5ee7a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0214c85c126c46599e8b3d8023c5ee7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } health_report { overall_health: HEALTHY } } }
I20260929 01:05:27.109772 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:27.106556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45242 (local address 127.0.98.1:40049)
0929 01:05:27.106692 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:27.106695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:27.106709 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:05:27.106782 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:27.106784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:27.106888 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:27.106985 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:05:27.107176 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.107695 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.108465 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:27.108698 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:27.108777 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:27.108981 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:05:27.109401 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:05:27.109488 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:27.109538 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 01:05:27.109589 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 01:05:27.113806 16735 raft_consensus.cc:1261] T eb55d4f12821418aa246761699f4cc22 P 676627090e2541f4b01d3612e2c0081d [term 1 FOLLOWER]: Refusing update from remote peer 0214c85c126c46599e8b3d8023c5ee7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:05:27.113806 16603 raft_consensus.cc:1261] T eb55d4f12821418aa246761699f4cc22 P a3c65a34ee9d4056833bec226605e720 [term 1 FOLLOWER]: Refusing update from remote peer 0214c85c126c46599e8b3d8023c5ee7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:05:27.114473 16517 consensus_queue.cc:1035] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:05:27.114668 16801 consensus_queue.cc:1035] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:05:27.123243 16805 mvcc.cc:205] Tried to move back new op lower bound from 7334477525451575296 to 7334477525359439872. Current Snapshot: MvccSnapshot[applied={T|T < 7334477525451575296}]
I20260929 01:05:27.130477 16322 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:05:27.130673 16322 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:05:27.133074 16322 catalog_manager.cc:5782] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd: Sending DeleteTablet for 3 replicas of tablet eb55d4f12821418aa246761699f4cc22
I20260929 01:05:27.133780 16715 tablet_service.cc:1514] Processing DeleteTablet for tablet eb55d4f12821418aa246761699f4cc22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:05:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43848
I20260929 01:05:27.133955 16583 tablet_service.cc:1514] Processing DeleteTablet for tablet eb55d4f12821418aa246761699f4cc22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:05:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34456
I20260929 01:05:27.134323 16814 tablet_replica.cc:330] T eb55d4f12821418aa246761699f4cc22 P 676627090e2541f4b01d3612e2c0081d: stopping tablet replica
I20260929 01:05:27.134403 16445 tablet_service.cc:1514] Processing DeleteTablet for tablet eb55d4f12821418aa246761699f4cc22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:05:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45236
I20260929 01:05:27.134573 16520 tablet_replica.cc:330] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a: stopping tablet replica
I20260929 01:05:27.134639 16815 tablet_replica.cc:330] T eb55d4f12821418aa246761699f4cc22 P a3c65a34ee9d4056833bec226605e720: stopping tablet replica
I20260929 01:05:27.134696 16520 raft_consensus.cc:2229] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:05:27.134954 16815 raft_consensus.cc:2229] T eb55d4f12821418aa246761699f4cc22 P a3c65a34ee9d4056833bec226605e720 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:05:27.135049 16520 raft_consensus.cc:2258] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:05:27.135255 16815 raft_consensus.cc:2258] T eb55d4f12821418aa246761699f4cc22 P a3c65a34ee9d4056833bec226605e720 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:05:27.135322 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 16361
I20260929 01:05:27.135011 16814 raft_consensus.cc:2229] T eb55d4f12821418aa246761699f4cc22 P 676627090e2541f4b01d3612e2c0081d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:05:27.136463 16814 raft_consensus.cc:2258] T eb55d4f12821418aa246761699f4cc22 P 676627090e2541f4b01d3612e2c0081d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:05:27.137897 16815 ts_tablet_manager.cc:1905] T eb55d4f12821418aa246761699f4cc22 P a3c65a34ee9d4056833bec226605e720: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:05:27.138090 16814 ts_tablet_manager.cc:1905] T eb55d4f12821418aa246761699f4cc22 P 676627090e2541f4b01d3612e2c0081d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:05:27.140748 16815 ts_tablet_manager.cc:1918] T eb55d4f12821418aa246761699f4cc22 P a3c65a34ee9d4056833bec226605e720: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:05:27.140758 16814 ts_tablet_manager.cc:1918] T eb55d4f12821418aa246761699f4cc22 P 676627090e2541f4b01d3612e2c0081d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:05:27.140857 16815 log.cc:1196] T eb55d4f12821418aa246761699f4cc22 P a3c65a34ee9d4056833bec226605e720: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/wal/wals/eb55d4f12821418aa246761699f4cc22
I20260929 01:05:27.140857 16814 log.cc:1196] T eb55d4f12821418aa246761699f4cc22 P 676627090e2541f4b01d3612e2c0081d: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/eb55d4f12821418aa246761699f4cc22
I20260929 01:05:27.141237 16814 ts_tablet_manager.cc:1939] T eb55d4f12821418aa246761699f4cc22 P 676627090e2541f4b01d3612e2c0081d: Deleting consensus metadata
I20260929 01:05:27.141237 16815 ts_tablet_manager.cc:1939] T eb55d4f12821418aa246761699f4cc22 P a3c65a34ee9d4056833bec226605e720: Deleting consensus metadata
W20260929 01:05:27.141987 16310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40049 (error 108)
I20260929 01:05:27.142493 16307 catalog_manager.cc:4841] TS a3c65a34ee9d4056833bec226605e720 (127.0.98.2:33785): tablet eb55d4f12821418aa246761699f4cc22 (table test-table [id=1893a630e02542f09fa8123855f335e2]) successfully deleted
I20260929 01:05:27.142493 16308 catalog_manager.cc:4841] TS 676627090e2541f4b01d3612e2c0081d (127.0.98.3:36387): tablet eb55d4f12821418aa246761699f4cc22 (table test-table [id=1893a630e02542f09fa8123855f335e2]) successfully deleted
I20260929 01:05:27.142982 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:27.142692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40049 (local address 127.0.0.1:45252)
0929 01:05:27.142822 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:27.142898 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 01:05:27.143258 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 16515
W20260929 01:05:27.143210 16310 catalog_manager.cc:4584] TS 0214c85c126c46599e8b3d8023c5ee7a (127.0.98.1:40049): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb55d4f12821418aa246761699f4cc22: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40049: connect: Connection refused (error 111)
I20260929 01:05:27.149294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 16652
I20260929 01:05:27.154889 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 16290
I20260929 01:05:47.161924 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42963
--webserver_interface=127.0.98.62
--webserver_port=33669
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:05:47.279621 16817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:47.279951 16817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:47.280035 16817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:47.284240 16817 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 01:05:47.284332 16817 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 01:05:47.284360 16817 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 01:05:47.284407 16817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:05:47.284447 16817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:47.284484 16817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:47.284519 16817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:05:47.284551 16817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:05:47.290194 16817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:05:47.291438 16817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:47.292511 16817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:47.301795 16823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:47.301895 16817 server_base.cc:1034] running on GCE node
W20260929 01:05:47.301797 16822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:47.301795 16825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:47.302642 16817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:47.303836 16817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:47.305001 16817 hybrid_clock.cc:648] HybridClock initialized: now 1790643947304981 us; error 33 us; skew 500 ppm
Sep 29 01:05:47 dist-test-slave-7zdl krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:47.312101 16817 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:05:47.313067 16817 webserver.cc:460] Webserver started at http://127.0.98.62:33669/ using document root <none> and password file <none>
I20260929 01:05:47.313704 16817 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:47.313769 16817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:47.317871 16817 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:05:47.320492 16832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:47.321861 16817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:05:47.321975 16817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "beea0aa0698d42feb60125ad540762dd"
format_stamp: "Formatted at 2026-09-29 01:05:05 on dist-test-slave-7zdl"
server_key: "2af0b61038c6efa396007027520720c0"
server_key_iv: "6c53a402d7ed9197f12089b407214405"
server_key_version: "encryptionkey@0"
I20260929 01:05:47.322634 16817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:47.346094 16817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:47.349555 16817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:47.349731 16817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:47.356820 16817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42963
I20260929 01:05:47.357120 16884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42963 every 8 connection(s)
I20260929 01:05:47.357966 16817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:05:47.359071 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 16817
I20260929 01:05:47.359495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40049
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42963
--webserver_port=37291
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:05:47.362915 16885 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:05:47.364521 16885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd: Bootstrap starting.
I20260929 01:05:47.384599 16885 log.cc:824] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:47.395603 16885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:05:47.396011 16885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd: Bootstrap complete.
I20260929 01:05:47.400141 16885 raft_consensus.cc:348] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beea0aa0698d42feb60125ad540762dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42963 } }
I20260929 01:05:47.400527 16885 raft_consensus.cc:729] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: beea0aa0698d42feb60125ad540762dd, State: Initialized, Role: FOLLOWER
I20260929 01:05:47.401058 16885 consensus_queue.cc:260] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beea0aa0698d42feb60125ad540762dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42963 } }
I20260929 01:05:47.401177 16885 raft_consensus.cc:388] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:05:47.401262 16885 raft_consensus.cc:482] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:05:47.401384 16885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:05:47.403342 16885 raft_consensus.cc:504] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beea0aa0698d42feb60125ad540762dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42963 } }
I20260929 01:05:47.403673 16885 leader_election.cc:302] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: beea0aa0698d42feb60125ad540762dd; no voters:
I20260929 01:05:47.403995 16885 leader_election.cc:288] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 01:05:47.404166 16889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [term 3 FOLLOWER]: Leader election won for term 3
I20260929 01:05:47.404438 16889 raft_consensus.cc:686] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [term 3 LEADER]: Becoming Leader. State: Replica: beea0aa0698d42feb60125ad540762dd, State: Running, Role: LEADER
I20260929 01:05:47.404752 16889 consensus_queue.cc:237] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beea0aa0698d42feb60125ad540762dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42963 } }
I20260929 01:05:47.405242 16885 sys_catalog.cc:564] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:05:47.406402 16891 sys_catalog.cc:455] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader beea0aa0698d42feb60125ad540762dd. Latest consensus state: current_term: 3 leader_uuid: "beea0aa0698d42feb60125ad540762dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beea0aa0698d42feb60125ad540762dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42963 } } }
I20260929 01:05:47.406533 16891 sys_catalog.cc:458] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [sys.catalog]: This master's current role is: LEADER
I20260929 01:05:47.406399 16890 sys_catalog.cc:455] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "beea0aa0698d42feb60125ad540762dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beea0aa0698d42feb60125ad540762dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42963 } } }
I20260929 01:05:47.406817 16890 sys_catalog.cc:458] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd [sys.catalog]: This master's current role is: LEADER
I20260929 01:05:47.407233 16898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:05:47.411734 16898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:05:47.412293 16898 catalog_manager.cc:1259] Loaded cluster ID: d9496da45080486e93791d0f00a5982c
I20260929 01:05:47.412375 16898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:05:47.414140 16898 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:05:47.414736 16898 catalog_manager.cc:5879] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd: Loaded TSK: 0
I20260929 01:05:47.414871 16898 catalog_manager.cc:5879] T 00000000000000000000000000000000 P beea0aa0698d42feb60125ad540762dd: Loaded TSK: 1
F20260929 01:05:47.415580 16898 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9ec1e8f9ad google::LogMessage::Fail() at ??:0
@ 0x7f9ec1e9288a google::LogMessage::SendToLog() at ??:0
@ 0x7f9ec1e8f4de google::LogMessage::Flush() at ??:0
@ 0x7f9ec1e90a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9ec4b5318d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f9ec4b4bf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f9ecc7f47bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f9ecc7ce99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f9ecc7cf178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f9ecc810df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f9ecc82f82b std::function<>::operator()() at ??:0
@ 0x7f9ecc7ceb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f9ecc7cff3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9ecc7cc579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f9ecc810640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9ec40e13d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9ec40e1c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f9ec40e33f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9ec40d2512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9ec57166db start_thread at ??:0
@ 0x7f9ec098d61f clone at ??:0
W20260929 01:05:47.499117 16887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:47.499366 16887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:47.499408 16887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:47.503850 16887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:47.503899 16887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:47.503962 16887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:05:47.510069 16887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37291
--tserver_master_addrs=127.0.98.62:42963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:05:47.511255 16887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:47.512334 16887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:47.522382 16935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:47.522384 16932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:47.522871 16887 server_base.cc:1034] running on GCE node
W20260929 01:05:47.522921 16933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:47.523676 16887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:47.524356 16887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:47.525571 16887 hybrid_clock.cc:648] HybridClock initialized: now 1790643947525539 us; error 50 us; skew 500 ppm
Sep 29 01:05:47 dist-test-slave-7zdl krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:47.528720 16887 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:05:47.529835 16887 webserver.cc:460] Webserver started at http://127.0.98.1:37291/ using document root <none> and password file <none>
I20260929 01:05:47.530575 16887 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:47.530669 16887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:47.535737 16887 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:05:47.538414 16942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:47.539947 16887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:05:47.540066 16887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "0214c85c126c46599e8b3d8023c5ee7a"
format_stamp: "Formatted at 2026-09-29 01:05:05 on dist-test-slave-7zdl"
server_key: "c29be331a1fdad92a67717e48aa5565d"
server_key_iv: "a9b993c47c0c6d6a2f6d27e51607d6a8"
server_key_version: "encryptionkey@0"
I20260929 01:05:47.540769 16887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:47.564623 16887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:47.568569 16887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:47.568794 16887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:47.569563 16887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:47.570938 16949 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 01:05:47.574959 16887 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 01:05:47.575032 16887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.001s
I20260929 01:05:47.575103 16887 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 01:05:47.577337 16887 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 01:05:47.577397 16887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 01:05:47.577754 16949 tablet_bootstrap.cc:492] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a: Bootstrap starting.
I20260929 01:05:47.588268 16887 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40049
I20260929 01:05:47.588521 17056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40049 every 8 connection(s)
I20260929 01:05:47.589541 16887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:05:47.597352 16949 log.cc:824] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a: Log is configured to *not* fsync() on all Append() calls
I20260929 01:05:47.597934 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 16887
I20260929 01:05:47.598407 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33785
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42963
--webserver_port=33617
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:05:47.604751 16949 tablet_bootstrap.cc:492] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:05:47.605118 16949 tablet_bootstrap.cc:492] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a: Bootstrap complete.
I20260929 01:05:47.606212 16949 ts_tablet_manager.cc:1397] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a: Time spent bootstrapping tablet: real 0.029s user 0.025s sys 0.000s
I20260929 01:05:47.608619 16949 raft_consensus.cc:348] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:05:47.609043 16949 raft_consensus.cc:729] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0214c85c126c46599e8b3d8023c5ee7a, State: Initialized, Role: FOLLOWER
I20260929 01:05:47.609515 16949 consensus_queue.cc:260] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:05:47.610468 16949 ts_tablet_manager.cc:1428] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
W20260929 01:05:47.685559 17059 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:05:47.591884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.1:53419)
0929 01:05:47.592492 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:47.592572 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:47.593669 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:47.684788 (+ 91119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":153,"threads_started":1}
W20260929 01:05:47.686318 17057 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42963 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42963: Client connection negotiation failed: client connection to 127.0.98.62:42963: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:05:47.687202 17059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:47.686908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.1:43095)
0929 01:05:47.687056 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:47.687134 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 01:05:47.687981 17059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:47.687728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.1:37689)
0929 01:05:47.687845 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:47.687905 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 01:05:47.688216 17057 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 01:05:47.737123 17061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:47.737404 17061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:47.737448 17061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:47.741509 17061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:47.741581 17061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:47.741640 17061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:05:47.747251 17061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33617
--tserver_master_addrs=127.0.98.62:42963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:05:47.748303 17061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:47.749285 17061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:47.758999 17068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:47.758975 17067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:47.758962 17070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:47.759392 17061 server_base.cc:1034] running on GCE node
I20260929 01:05:47.759804 17061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:47.760445 17061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:47.761601 17061 hybrid_clock.cc:648] HybridClock initialized: now 1790643947761584 us; error 32 us; skew 500 ppm
Sep 29 01:05:47 dist-test-slave-7zdl krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:47.764344 17061 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:05:47.765240 17061 webserver.cc:460] Webserver started at http://127.0.98.2:33617/ using document root <none> and password file <none>
I20260929 01:05:47.765754 17061 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:47.765811 17061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:47.769528 17061 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:05:47.771641 17077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:47.772873 17061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:05:47.772991 17061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "a3c65a34ee9d4056833bec226605e720"
format_stamp: "Formatted at 2026-09-29 01:05:05 on dist-test-slave-7zdl"
server_key: "ed5133ca58efdc874b66c594e7f6d1ad"
server_key_iv: "c5dc2b35e71c5a1635c517166b7cda71"
server_key_version: "encryptionkey@0"
I20260929 01:05:47.773516 17061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:47.789012 17061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:47.792261 17061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:47.792471 17061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:47.793087 17061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:47.793927 17061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:47.793974 17061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:47.794014 17061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:47.794063 17061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:47.803594 17061 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33785
I20260929 01:05:47.803611 17190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33785 every 8 connection(s)
I20260929 01:05:47.804699 17061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:05:47.808135 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 17061
I20260929 01:05:47.808673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36387
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42963
--webserver_port=36393
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:05:47.809338 17193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:47.807009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.2:49853)
0929 01:05:47.807491 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:47.808246 (+ 755us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":110,"threads_started":1}
W20260929 01:05:47.809892 17191 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42963 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42963: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
I20260929 01:05:47.810856 17193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:47.810434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.2:33131)
0929 01:05:47.810686 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:47.810742 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260929 01:05:47.811736 17193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:47.811425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.2:58351)
0929 01:05:47.811540 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:47.811655 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 01:05:47.811926 17191 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 01:05:47.925482 17194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:05:47.925735 17194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:05:47.925804 17194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:05:47.929809 17194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:05:47.929867 17194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:05:47.929926 17194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:05:47.935349 17194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36393
--tserver_master_addrs=127.0.98.62:42963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:05:47.936455 17194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:05:47.937486 17194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:05:47.947239 17200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:47.947243 17199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:05:47.947243 17202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:05:47.947559 17194 server_base.cc:1034] running on GCE node
I20260929 01:05:47.947976 17194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:05:47.948664 17194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:05:47.949836 17194 hybrid_clock.cc:648] HybridClock initialized: now 1790643947949799 us; error 47 us; skew 500 ppm
Sep 29 01:05:47 dist-test-slave-7zdl krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:05:47.952626 17194 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:05:47.953516 17194 webserver.cc:460] Webserver started at http://127.0.98.3:36393/ using document root <none> and password file <none>
I20260929 01:05:47.954136 17194 fs_manager.cc:359] Metadata directory not provided
I20260929 01:05:47.954201 17194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:05:47.958161 17194 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:05:47.960264 17209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:47.961462 17194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260929 01:05:47.961591 17194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "676627090e2541f4b01d3612e2c0081d"
format_stamp: "Formatted at 2026-09-29 01:05:05 on dist-test-slave-7zdl"
server_key: "196bdf62a98d81ef4b64a11376058ea1"
server_key_iv: "dd92bd6c1340eae2bcc14d048a64bdf7"
server_key_version: "encryptionkey@0"
I20260929 01:05:47.962158 17194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:05:47.974138 17194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:05:47.977188 17194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:05:47.977458 17194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:05:47.978093 17194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:05:47.979001 17194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:05:47.979074 17194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:47.979143 17194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:05:47.979200 17194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:05:47.988453 17194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36387
I20260929 01:05:47.988472 17322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36387 every 8 connection(s)
I20260929 01:05:47.989499 17194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:05:47.993155 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:47.991674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.3:50059)
0929 01:05:47.992333 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:47.992445 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":102,"threads_started":1}
W20260929 01:05:47.993690 17323 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42963 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42963: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
I20260929 01:05:47.994244 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:47.994041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.3:46125)
0929 01:05:47.994147 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:47.994191 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 01:05:47.994937 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:47.994703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.3:34175)
0929 01:05:47.994840 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:47.994893 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 01:05:47.995111 17323 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 01:05:47.996349 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 17194
I20260929 01:05:48.689486 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:48.688963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.1:50807)
0929 01:05:48.689290 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:48.689363 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":82,"threads_started":1}
I20260929 01:05:48.812855 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:48.812456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.2:54839)
0929 01:05:48.812686 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:48.812745 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":73,"threads_started":1}
I20260929 01:05:48.996152 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:48.995730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.3:53783)
0929 01:05:48.995988 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:48.996048 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
I20260929 01:05:49.016736 17330 raft_consensus.cc:482] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:05:49.016978 17330 raft_consensus.cc:504] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:05:49.018186 17330 leader_election.cc:288] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 676627090e2541f4b01d3612e2c0081d (127.0.98.3:36387), a3c65a34ee9d4056833bec226605e720 (127.0.98.2:33785)
Sep 29 01:05:49 dist-test-slave-7zdl krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 29 01:05:49 dist-test-slave-7zdl krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260929 01:05:49.032518 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.018455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33785 (local address 127.0.98.1:57741)
0929 01:05:49.018835 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:49.018852 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:49.018978 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:49.020721 (+ 1743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:49.020727 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:49.021280 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:05:49.021720 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:49.021742 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.023088 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.023091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:49.024029 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:49.024039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.024783 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.024787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:49.024997 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:49.025720 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:05:49.025733 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:05:49.027701 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:05:49.029923 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:49.029925 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:49.029927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:49.030355 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:49.030760 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:49.030764 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:49.030766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:49.030973 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:49.031599 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:05:49.031604 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:05:49.031831 (+ 227us) client_negotiation.cc:769] Sending connection context
0929 01:05:49.032014 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 01:05:49.032319 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":144,"spinlock_wait_cycles":36992,"thread_start_us":74,"threads_started":1}
I20260929 01:05:49.032578 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.018683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57741 (local address 127.0.98.2:33785)
0929 01:05:49.019165 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:49.019194 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:49.019226 (+ 32us) server_negotiation.cc:407] Connection header received
0929 01:05:49.020015 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:49.020022 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:49.020362 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:49.020668 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:49.021904 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.022939 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.024168 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.024649 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.024820 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:49.027895 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:49.027904 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:49.027922 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:49.027961 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:49.029787 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:49.030483 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:49.030489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:49.030495 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:49.030648 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:49.031079 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:49.031082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:49.031083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:49.031498 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:49.031603 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:49.032144 (+ 541us) server_negotiation.cc:299] Negotiation successful
0929 01:05:49.032410 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":152,"threads_started":1}
I20260929 01:05:49.032967 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.018562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46661 (local address 127.0.98.3:36387)
0929 01:05:49.019087 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0929 01:05:49.019111 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 01:05:49.019171 (+ 60us) server_negotiation.cc:407] Connection header received
0929 01:05:49.020001 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:05:49.020008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:05:49.020362 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:05:49.020668 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:05:49.021904 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.022941 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.024179 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.024644 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.024827 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:49.027582 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:05:49.027588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:05:49.027601 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:05:49.027628 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:05:49.029723 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:49.030498 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:49.030504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:49.030511 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:49.030658 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:05:49.031643 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:05:49.031646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:05:49.031647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:05:49.032014 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:05:49.032157 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 01:05:49.032485 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 01:05:49.032782 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":55,"threads_started":1}
I20260929 01:05:49.032830 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.018418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36387 (local address 127.0.98.1:46661)
0929 01:05:49.018623 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:49.018638 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:05:49.018748 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:05:49.020722 (+ 1974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:05:49.020727 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:05:49.021280 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:05:49.021720 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:49.021735 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.023088 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.023097 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:49.024031 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:05:49.024040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:05:49.024783 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:05:49.024787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:05:49.025003 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:05:49.025658 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:05:49.025675 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:05:49.027357 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:05:49.029883 (+ 2526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:49.029889 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:49.029910 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:49.030346 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:49.030829 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:05:49.030834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:05:49.030838 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:05:49.030973 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:05:49.032128 (+ 1155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:05:49.032131 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:05:49.032232 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 01:05:49.032341 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 01:05:49.032476 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":83,"spinlock_wait_cycles":5376}
W20260929 01:05:49.035537 16944 leader_election.cc:341] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 676627090e2541f4b01d3612e2c0081d (127.0.98.3:36387): Not found: Tablet not found: eb55d4f12821418aa246761699f4cc22
W20260929 01:05:49.035754 16943 leader_election.cc:341] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3c65a34ee9d4056833bec226605e720 (127.0.98.2:33785): Not found: Tablet not found: eb55d4f12821418aa246761699f4cc22
I20260929 01:05:49.035804 16943 leader_election.cc:302] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0214c85c126c46599e8b3d8023c5ee7a; no voters: 676627090e2541f4b01d3612e2c0081d, a3c65a34ee9d4056833bec226605e720
I20260929 01:05:49.036055 17330 raft_consensus.cc:2726] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:05:49.690891 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.690350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.1:53867)
0929 01:05:49.690668 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:49.690726 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":117,"threads_started":1}
I20260929 01:05:49.813963 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.813576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.2:60487)
0929 01:05:49.813816 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:49.813872 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":75,"threads_started":1}
I20260929 01:05:49.997314 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:49.996885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.3:38191)
0929 01:05:49.997165 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:49.997227 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":74,"threads_started":1}
I20260929 01:05:50.692070 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:50.691647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.1:55767)
0929 01:05:50.691900 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:50.691963 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260929 01:05:50.815128 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:50.814705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.2:47373)
0929 01:05:50.814964 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:50.815018 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":91,"threads_started":1}
I20260929 01:05:50.834038 17339 raft_consensus.cc:482] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:05:50.834172 17339 raft_consensus.cc:504] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:05:50.834663 17339 leader_election.cc:288] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 676627090e2541f4b01d3612e2c0081d (127.0.98.3:36387), a3c65a34ee9d4056833bec226605e720 (127.0.98.2:33785)
W20260929 01:05:50.835412 16943 leader_election.cc:341] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3c65a34ee9d4056833bec226605e720 (127.0.98.2:33785): Not found: Tablet not found: eb55d4f12821418aa246761699f4cc22
W20260929 01:05:50.835516 16944 leader_election.cc:341] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 676627090e2541f4b01d3612e2c0081d (127.0.98.3:36387): Not found: Tablet not found: eb55d4f12821418aa246761699f4cc22
I20260929 01:05:50.835690 16944 leader_election.cc:302] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0214c85c126c46599e8b3d8023c5ee7a; no voters: 676627090e2541f4b01d3612e2c0081d, a3c65a34ee9d4056833bec226605e720
I20260929 01:05:50.835839 17339 raft_consensus.cc:2726] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:05:50.998661 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:50.998210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.3:43541)
0929 01:05:50.998494 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:50.998554 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":117,"threads_started":1}
I20260929 01:05:51.693300 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:51.692888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.1:49411)
0929 01:05:51.693137 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:51.693200 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":72,"threads_started":1}
I20260929 01:05:51.816251 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:51.815854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.2:59875)
0929 01:05:51.816106 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:51.816165 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":100,"threads_started":1}
I20260929 01:05:51.999823 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:51.999388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.3:39231)
0929 01:05:51.999639 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:51.999693 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":71,"threads_started":1}
I20260929 01:05:52.694713 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:52.694236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.1:41823)
0929 01:05:52.694558 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:52.694618 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":68,"threads_started":1}
I20260929 01:05:52.817348 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:52.816976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.2:58239)
0929 01:05:52.817203 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:52.817255 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":71,"threads_started":1}
I20260929 01:05:52.907618 17346 raft_consensus.cc:482] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:05:52.907753 17346 raft_consensus.cc:504] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:05:52.908265 17346 leader_election.cc:288] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 676627090e2541f4b01d3612e2c0081d (127.0.98.3:36387), a3c65a34ee9d4056833bec226605e720 (127.0.98.2:33785)
W20260929 01:05:52.909049 16944 leader_election.cc:341] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 676627090e2541f4b01d3612e2c0081d (127.0.98.3:36387): Not found: Tablet not found: eb55d4f12821418aa246761699f4cc22
W20260929 01:05:52.909221 16943 leader_election.cc:341] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3c65a34ee9d4056833bec226605e720 (127.0.98.2:33785): Not found: Tablet not found: eb55d4f12821418aa246761699f4cc22
I20260929 01:05:52.909307 16943 leader_election.cc:302] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0214c85c126c46599e8b3d8023c5ee7a; no voters: 676627090e2541f4b01d3612e2c0081d, a3c65a34ee9d4056833bec226605e720
I20260929 01:05:52.909451 17346 raft_consensus.cc:2726] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:05:53.001025 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:53.000641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.3:50059)
0929 01:05:53.000883 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:53.000937 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":71,"threads_started":1}
I20260929 01:05:53.695984 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:53.695540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.1:41605)
0929 01:05:53.695829 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:53.695883 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":112,"threads_started":1}
I20260929 01:05:53.818439 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:53.818058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.2:47619)
0929 01:05:53.818303 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:53.818355 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
I20260929 01:05:54.002327 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:54.001820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.3:47329)
0929 01:05:54.002148 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:54.002215 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":72,"threads_started":1}
I20260929 01:05:54.697127 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:54.696731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.1:35755)
0929 01:05:54.696968 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:54.697020 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
I20260929 01:05:54.819787 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:54.819237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.2:41071)
0929 01:05:54.819536 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:54.819665 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":90,"threads_started":1}
I20260929 01:05:55.003533 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:55.003103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.3:45553)
0929 01:05:55.003393 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:55.003444 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":83,"threads_started":1}
I20260929 01:05:55.698522 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:55.697992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.1:56361)
0929 01:05:55.698370 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:55.698428 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":115,"threads_started":1}
I20260929 01:05:55.821002 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:55.820506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.2:38561)
0929 01:05:55.820856 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:55.820911 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":87,"threads_started":1}
I20260929 01:05:56.004601 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:56.004213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.3:55049)
0929 01:05:56.004462 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:56.004515 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
I20260929 01:05:56.699951 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:56.699476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.1:33861)
0929 01:05:56.699791 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:56.699855 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":104,"threads_started":1}
I20260929 01:05:56.822358 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:56.821904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.2:52707)
0929 01:05:56.822200 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:56.822266 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":93,"threads_started":1}
I20260929 01:05:57.005993 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:57.005498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.3:46617)
0929 01:05:57.005824 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:57.005889 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":108,"threads_started":1}
I20260929 01:05:57.684535 17360 raft_consensus.cc:482] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:05:57.684702 17360 raft_consensus.cc:504] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:05:57.685123 17360 leader_election.cc:288] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 676627090e2541f4b01d3612e2c0081d (127.0.98.3:36387), a3c65a34ee9d4056833bec226605e720 (127.0.98.2:33785)
W20260929 01:05:57.685837 16944 leader_election.cc:341] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 676627090e2541f4b01d3612e2c0081d (127.0.98.3:36387): Not found: Tablet not found: eb55d4f12821418aa246761699f4cc22
W20260929 01:05:57.685990 16943 leader_election.cc:341] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3c65a34ee9d4056833bec226605e720 (127.0.98.2:33785): Not found: Tablet not found: eb55d4f12821418aa246761699f4cc22
I20260929 01:05:57.686079 16943 leader_election.cc:302] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0214c85c126c46599e8b3d8023c5ee7a; no voters: 676627090e2541f4b01d3612e2c0081d, a3c65a34ee9d4056833bec226605e720
I20260929 01:05:57.686251 17360 raft_consensus.cc:2726] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:05:57.701135 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:57.700781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.1:59773)
0929 01:05:57.701001 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:57.701051 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":66,"threads_started":1}
I20260929 01:05:57.823748 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:57.823237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.2:37443)
0929 01:05:57.823551 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:57.823615 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":91,"threads_started":1}
I20260929 01:05:58.007228 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:58.006861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.3:52479)
0929 01:05:58.007090 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:58.007139 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
I20260929 01:05:58.702337 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:58.701867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.1:49281)
0929 01:05:58.702175 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:58.702242 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":91,"threads_started":1}
I20260929 01:05:58.824811 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:58.824419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.2:48305)
0929 01:05:58.824663 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:58.824715 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":89,"threads_started":1}
I20260929 01:05:59.008358 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:59.007905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.3:51741)
0929 01:05:59.008211 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:59.008266 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":72,"threads_started":1}
I20260929 01:05:59.703645 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:59.703248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.1:41819)
0929 01:05:59.703484 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:59.703536 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":80,"threads_started":1}
I20260929 01:05:59.825980 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:05:59.825522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.2:42801)
0929 01:05:59.825823 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 01:05:59.825882 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":114,"threads_started":1}
I20260929 01:06:00.009537 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:00.009117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.3:45487)
0929 01:06:00.009388 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:00.009443 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":83,"threads_started":1}
I20260929 01:06:00.704924 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:00.704499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.1:36389)
0929 01:06:00.704769 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:00.704822 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":83,"threads_started":1}
I20260929 01:06:00.827209 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:00.826767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.2:57931)
0929 01:06:00.827061 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:00.827114 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":80,"threads_started":1}
I20260929 01:06:01.010885 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:01.010366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.3:44007)
0929 01:06:01.010629 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:01.010684 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":74,"threads_started":1}
I20260929 01:06:01.706436 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:01.705751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.1:34947)
0929 01:06:01.706193 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:01.706277 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":119,"threads_started":1}
I20260929 01:06:01.828317 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:01.827906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.2:45909)
0929 01:06:01.828165 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:01.828223 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":72,"threads_started":1}
I20260929 01:06:02.012207 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:02.011730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.3:58185)
0929 01:06:02.011978 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:02.012045 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":69,"threads_started":1}
I20260929 01:06:02.231704 17376 raft_consensus.cc:482] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:06:02.231838 17376 raft_consensus.cc:504] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "676627090e2541f4b01d3612e2c0081d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36387 } } peers { permanent_uuid: "a3c65a34ee9d4056833bec226605e720" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33785 } } peers { permanent_uuid: "0214c85c126c46599e8b3d8023c5ee7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40049 } }
I20260929 01:06:02.232213 17376 leader_election.cc:288] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 676627090e2541f4b01d3612e2c0081d (127.0.98.3:36387), a3c65a34ee9d4056833bec226605e720 (127.0.98.2:33785)
W20260929 01:06:02.232985 16943 leader_election.cc:341] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3c65a34ee9d4056833bec226605e720 (127.0.98.2:33785): Not found: Tablet not found: eb55d4f12821418aa246761699f4cc22
W20260929 01:06:02.233125 16944 leader_election.cc:341] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 676627090e2541f4b01d3612e2c0081d (127.0.98.3:36387): Not found: Tablet not found: eb55d4f12821418aa246761699f4cc22
I20260929 01:06:02.233222 16944 leader_election.cc:302] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0214c85c126c46599e8b3d8023c5ee7a; no voters: 676627090e2541f4b01d3612e2c0081d, a3c65a34ee9d4056833bec226605e720
I20260929 01:06:02.233428 17376 raft_consensus.cc:2726] T eb55d4f12821418aa246761699f4cc22 P 0214c85c126c46599e8b3d8023c5ee7a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:06:02.708055 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:02.707554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.1:55985)
0929 01:06:02.707849 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:02.707921 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":91,"threads_started":1}
I20260929 01:06:02.829618 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:02.829156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.2:58621)
0929 01:06:02.829471 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:02.829526 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":72,"threads_started":1}
I20260929 01:06:03.004900 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 16887
I20260929 01:06:03.010514 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 17061
I20260929 01:06:03.013504 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:03.013100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42963 (local address 127.0.98.3:57317)
0929 01:06:03.013364 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:03.013415 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":90,"threads_started":1}
I20260929 01:06:03.016249 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 17194
2026-09-29T01:06:03Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59260 ms)
[----------] 30 tests from SecurityITest (242122 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:03 dist-test-slave-7zdl krb5kdc[17382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:03 dist-test-slave-7zdl krb5kdc[17382](info): set up 2 sockets
Sep 29 01:06:03 dist-test-slave-7zdl krb5kdc[17382](info): commencing operation
krb5kdc: starting...
W20260929 01:06:04.058454 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:06:04 dist-test-slave-7zdl krb5kdc[17382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:06:04.178444 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43525
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:04.285560 17395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:04.285851 17395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:04.285924 17395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:04.289922 17395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:04.290023 17395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:04.290059 17395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:04.290108 17395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:04.290143 17395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:04.295256 17395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:04.296288 17395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:04.297300 17395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:04.305631 17403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:04.305667 17401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:04.305781 17395 server_base.cc:1034] running on GCE node
W20260929 01:06:04.305667 17400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:04.306388 17395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:04.307469 17395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:04.308624 17395 hybrid_clock.cc:648] HybridClock initialized: now 1790643964308600 us; error 42 us; skew 500 ppm
Sep 29 01:06:04 dist-test-slave-7zdl krb5kdc[17382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:04.311530 17395 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:06:04.312431 17395 webserver.cc:460] Webserver started at http://127.0.98.62:41011/ using document root <none> and password file <none>
I20260929 01:06:04.312961 17395 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:04.313048 17395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:04.313298 17395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:04.315308 17395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "37e84fdbfe4c419094bc8f6d7c9f9c68"
format_stamp: "Formatted at 2026-09-29 01:06:04 on dist-test-slave-7zdl"
server_key: "9ceff94c3de545de8da2ca9bba891334"
server_key_iv: "f4730b3aeb4a9ce75a0f6cf8874f8e27"
server_key_version: "encryptionkey@0"
I20260929 01:06:04.315860 17395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "37e84fdbfe4c419094bc8f6d7c9f9c68"
format_stamp: "Formatted at 2026-09-29 01:06:04 on dist-test-slave-7zdl"
server_key: "9ceff94c3de545de8da2ca9bba891334"
server_key_iv: "f4730b3aeb4a9ce75a0f6cf8874f8e27"
server_key_version: "encryptionkey@0"
I20260929 01:06:04.319375 17395 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:06:04.321590 17410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:04.322921 17395 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:06:04.323130 17395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "37e84fdbfe4c419094bc8f6d7c9f9c68"
format_stamp: "Formatted at 2026-09-29 01:06:04 on dist-test-slave-7zdl"
server_key: "9ceff94c3de545de8da2ca9bba891334"
server_key_iv: "f4730b3aeb4a9ce75a0f6cf8874f8e27"
server_key_version: "encryptionkey@0"
I20260929 01:06:04.323253 17395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:04.336668 17395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:04.339465 17395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:04.339682 17395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:04.346640 17395 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43525
I20260929 01:06:04.346657 17462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43525 every 8 connection(s)
I20260929 01:06:04.347554 17395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:04.350139 17463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:04.354053 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 17395
I20260929 01:06:04.354178 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:04.354460 392 external_mini_cluster.cc:1428] Setting key b6c5d36617cf6ff4a788e0b190a3391e
I20260929 01:06:04.356762 17463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37e84fdbfe4c419094bc8f6d7c9f9c68: Bootstrap starting.
I20260929 01:06:04.359476 17463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37e84fdbfe4c419094bc8f6d7c9f9c68: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:04.360323 17463 log.cc:824] T 00000000000000000000000000000000 P 37e84fdbfe4c419094bc8f6d7c9f9c68: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:04.362494 17463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37e84fdbfe4c419094bc8f6d7c9f9c68: No bootstrap required, opened a new log
Sep 29 01:06:04 dist-test-slave-7zdl krb5kdc[17382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:06:04.366622 17463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37e84fdbfe4c419094bc8f6d7c9f9c68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e84fdbfe4c419094bc8f6d7c9f9c68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43525 } }
I20260929 01:06:04.366931 17463 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37e84fdbfe4c419094bc8f6d7c9f9c68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:04.366990 17463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37e84fdbfe4c419094bc8f6d7c9f9c68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37e84fdbfe4c419094bc8f6d7c9f9c68, State: Initialized, Role: FOLLOWER
I20260929 01:06:04.367478 17463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37e84fdbfe4c419094bc8f6d7c9f9c68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e84fdbfe4c419094bc8f6d7c9f9c68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43525 } }
I20260929 01:06:04.367612 17463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37e84fdbfe4c419094bc8f6d7c9f9c68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:04.367695 17463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37e84fdbfe4c419094bc8f6d7c9f9c68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:04.367786 17463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37e84fdbfe4c419094bc8f6d7c9f9c68 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:04.368803 17463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37e84fdbfe4c419094bc8f6d7c9f9c68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e84fdbfe4c419094bc8f6d7c9f9c68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43525 } }
I20260929 01:06:04.369165 17463 leader_election.cc:302] T 00000000000000000000000000000000 P 37e84fdbfe4c419094bc8f6d7c9f9c68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37e84fdbfe4c419094bc8f6d7c9f9c68; no voters:
I20260929 01:06:04.369503 17463 leader_election.cc:288] T 00000000000000000000000000000000 P 37e84fdbfe4c419094bc8f6d7c9f9c68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:04.369697 17468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37e84fdbfe4c419094bc8f6d7c9f9c68 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:04.369980 17468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37e84fdbfe4c419094bc8f6d7c9f9c68 [term 1 LEADER]: Becoming Leader. State: Replica: 37e84fdbfe4c419094bc8f6d7c9f9c68, State: Running, Role: LEADER
I20260929 01:06:04.370265 17468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37e84fdbfe4c419094bc8f6d7c9f9c68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e84fdbfe4c419094bc8f6d7c9f9c68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43525 } }
I20260929 01:06:04.370530 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:04.355848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36458 (local address 127.0.98.62:43525)
0929 01:06:04.356298 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:04.356317 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:04.356360 (+ 43us) server_negotiation.cc:407] Connection header received
0929 01:06:04.357182 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:04.357204 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:04.357610 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:04.357983 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:04.358624 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:04.360144 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:04.361215 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:04.361868 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:04.362025 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:04.364648 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:04.364654 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:04.364671 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:04.364701 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:04.367323 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:04.367807 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:04.367812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:04.367816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:04.367970 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:04.368284 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:04.368287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:04.368288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:04.368647 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:04.368814 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:04.369114 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 01:06:04.369401 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":171,"threads_started":1}
I20260929 01:06:04.370786 17463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37e84fdbfe4c419094bc8f6d7c9f9c68 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:04.371435 17468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37e84fdbfe4c419094bc8f6d7c9f9c68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37e84fdbfe4c419094bc8f6d7c9f9c68. Latest consensus state: current_term: 1 leader_uuid: "37e84fdbfe4c419094bc8f6d7c9f9c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e84fdbfe4c419094bc8f6d7c9f9c68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43525 } } }
I20260929 01:06:04.371994 17469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37e84fdbfe4c419094bc8f6d7c9f9c68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37e84fdbfe4c419094bc8f6d7c9f9c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e84fdbfe4c419094bc8f6d7c9f9c68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43525 } } }
I20260929 01:06:04.372112 17468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37e84fdbfe4c419094bc8f6d7c9f9c68 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:04.372145 17469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37e84fdbfe4c419094bc8f6d7c9f9c68 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:04.372965 17473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:04.375934 17473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:04.381299 17473 catalog_manager.cc:1347] Generated new cluster ID: 52e97724007345a3a0d21e8846e72ab4
I20260929 01:06:04.381402 17473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:04.399461 17473 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:04.400323 17473 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:04.410774 17473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37e84fdbfe4c419094bc8f6d7c9f9c68: Generated new TSK 0
I20260929 01:06:04.411450 17473 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 01:06:04.446271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43525
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:04.561576 17489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:04.561841 17489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:04.561884 17489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:04.565948 17489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:04.566002 17489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:04.566057 17489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:06:04.571341 17489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:04.572342 17489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:04.573375 17489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:04.582770 17497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:04.582872 17495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:04.582746 17494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:04.582975 17489 server_base.cc:1034] running on GCE node
I20260929 01:06:04.583544 17489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:04.584162 17489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:04.585355 17489 hybrid_clock.cc:648] HybridClock initialized: now 1790643964585315 us; error 49 us; skew 500 ppm
Sep 29 01:06:04 dist-test-slave-7zdl krb5kdc[17382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:04.588208 17489 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:06:04.589071 17489 webserver.cc:460] Webserver started at http://127.0.98.1:34349/ using document root <none> and password file <none>
I20260929 01:06:04.589617 17489 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:04.589704 17489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:04.589926 17489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:04.592027 17489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "f83e6939f5074c7aa4d3b2d12d31871e"
format_stamp: "Formatted at 2026-09-29 01:06:04 on dist-test-slave-7zdl"
server_key: "2434734be60af341a31dc7b6eeda1665"
server_key_iv: "5e4b2b2e88289cc089d5115c48d22f65"
server_key_version: "encryptionkey@0"
I20260929 01:06:04.592640 17489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f83e6939f5074c7aa4d3b2d12d31871e"
format_stamp: "Formatted at 2026-09-29 01:06:04 on dist-test-slave-7zdl"
server_key: "2434734be60af341a31dc7b6eeda1665"
server_key_iv: "5e4b2b2e88289cc089d5115c48d22f65"
server_key_version: "encryptionkey@0"
I20260929 01:06:04.596251 17489 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:06:04.598591 17504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:04.599875 17489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:06:04.600004 17489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "f83e6939f5074c7aa4d3b2d12d31871e"
format_stamp: "Formatted at 2026-09-29 01:06:04 on dist-test-slave-7zdl"
server_key: "2434734be60af341a31dc7b6eeda1665"
server_key_iv: "5e4b2b2e88289cc089d5115c48d22f65"
server_key_version: "encryptionkey@0"
I20260929 01:06:04.600142 17489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:04.617792 17489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:04.620918 17489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:04.621125 17489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:04.621711 17489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:06:04.622579 17489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:06:04.622649 17489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:04.622726 17489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:06:04.622785 17489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:04.632189 17489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33157
I20260929 01:06:04.632300 17617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33157 every 8 connection(s)
I20260929 01:06:04.633134 17489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
Sep 29 01:06:04 dist-test-slave-7zdl krb5kdc[17382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:06:04.643332 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 17489
I20260929 01:06:04.643499 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:06:04.643800 392 external_mini_cluster.cc:1428] Setting key 0e1e5961cc20d96b8937ed9cc4f03c4f
I20260929 01:06:04.647024 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:04.634924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58081 (local address 127.0.98.62:43525)
0929 01:06:04.635055 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:04.635059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:04.635881 (+ 822us) server_negotiation.cc:407] Connection header received
0929 01:06:04.637227 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:04.637230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:04.637290 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:04.637400 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:04.638525 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:04.639126 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:04.640433 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:04.640678 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:04.640793 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:04.643285 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:04.643294 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:04.643296 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:04.643318 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:04.644889 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:04.645522 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:04.645525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:04.645527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:04.645609 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:04.645953 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:04.645958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:04.645960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:04.646138 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:04.646241 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:04.646720 (+ 479us) server_negotiation.cc:299] Negotiation successful
0929 01:06:04.646863 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:06:04.647910 17620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:04.635316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43525 (local address 127.0.98.1:58081)
0929 01:06:04.635732 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:04.635773 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:04.636923 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:04.637569 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:04.637576 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:04.637912 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:04.638373 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:04.638385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:04.639326 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:04.639330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:04.640276 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:04.640286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:04.640828 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:04.640833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:04.641008 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:04.641613 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:04.641628 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:04.643121 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:04.645026 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:04.645032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:04.645046 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:04.645405 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:04.645699 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:04.645703 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:04.645712 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:04.645858 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:04.646239 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:04.646245 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:04.646485 (+ 240us) client_negotiation.cc:769] Sending connection context
0929 01:06:04.646666 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 01:06:04.647033 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":100,"threads_started":1}
I20260929 01:06:04.649187 17618 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43525
I20260929 01:06:04.649475 17618 heartbeater.cc:464] Registering TS with master...
I20260929 01:06:04.650120 17618 heartbeater.cc:511] Master 127.0.98.62:43525 requested a full tablet report, sending...
I20260929 01:06:04.651739 17427 ts_manager.cc:194] Registered new tserver with Master: f83e6939f5074c7aa4d3b2d12d31871e (127.0.98.1:33157)
I20260929 01:06:04.652849 17427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58081
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 01:06:04.670258 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43525
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:04.781924 17623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:04.782204 17623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:04.782284 17623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:04.786405 17623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:04.786490 17623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:04.786582 17623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:06:04.791779 17623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:04.792788 17623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:04.793790 17623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:04.803217 17628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:04.803277 17631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:04.803288 17629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:04.803381 17623 server_base.cc:1034] running on GCE node
I20260929 01:06:04.803908 17623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:04.804556 17623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:04.805711 17623 hybrid_clock.cc:648] HybridClock initialized: now 1790643964805695 us; error 27 us; skew 500 ppm
Sep 29 01:06:04 dist-test-slave-7zdl krb5kdc[17382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:04.808568 17623 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:06:04.809448 17623 webserver.cc:460] Webserver started at http://127.0.98.2:37991/ using document root <none> and password file <none>
I20260929 01:06:04.810014 17623 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:04.810103 17623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:04.810329 17623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:04.812336 17623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "c3dd4fe49cd2475495299b59666b4986"
format_stamp: "Formatted at 2026-09-29 01:06:04 on dist-test-slave-7zdl"
server_key: "6530ce6e9029ef48d9e5e929f58210a8"
server_key_iv: "90333bc46d8b725ec41716dcffadc25e"
server_key_version: "encryptionkey@0"
I20260929 01:06:04.812903 17623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c3dd4fe49cd2475495299b59666b4986"
format_stamp: "Formatted at 2026-09-29 01:06:04 on dist-test-slave-7zdl"
server_key: "6530ce6e9029ef48d9e5e929f58210a8"
server_key_iv: "90333bc46d8b725ec41716dcffadc25e"
server_key_version: "encryptionkey@0"
I20260929 01:06:04.816509 17623 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:06:04.818903 17638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:04.820051 17623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:06:04.820181 17623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "c3dd4fe49cd2475495299b59666b4986"
format_stamp: "Formatted at 2026-09-29 01:06:04 on dist-test-slave-7zdl"
server_key: "6530ce6e9029ef48d9e5e929f58210a8"
server_key_iv: "90333bc46d8b725ec41716dcffadc25e"
server_key_version: "encryptionkey@0"
I20260929 01:06:04.820327 17623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:04.850039 17623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:04.853191 17623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:04.853403 17623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:04.854012 17623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:06:04.854866 17623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:06:04.854936 17623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:04.855003 17623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:06:04.855053 17623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:04.863667 17623 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43569
I20260929 01:06:04.863698 17751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43569 every 8 connection(s)
I20260929 01:06:04.864548 17623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:06:04.866045 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 17623
I20260929 01:06:04.866187 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:06:04.866504 392 external_mini_cluster.cc:1428] Setting key 4f1ae444ba03c562f3cfc303dfa83a82
Sep 29 01:06:04 dist-test-slave-7zdl krb5kdc[17382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 01:06:04.879802 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:04.866353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47105 (local address 127.0.98.62:43525)
0929 01:06:04.866510 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:04.866515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:04.867507 (+ 992us) server_negotiation.cc:407] Connection header received
0929 01:06:04.868937 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:04.868944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:04.869010 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:04.869103 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:04.870269 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:04.871026 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:04.872899 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:04.873244 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:04.873360 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:04.876156 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:04.876159 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:04.876161 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:04.876183 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:04.877609 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:04.878175 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:04.878180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:04.878182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:04.878261 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:04.878638 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:04.878646 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:04.878648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:04.878873 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:04.879043 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:04.879581 (+ 538us) server_negotiation.cc:299] Negotiation successful
0929 01:06:04.879680 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 01:06:04.880985 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:04.866843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43525 (local address 127.0.98.2:47105)
0929 01:06:04.867359 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:04.867414 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:04.868723 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:04.869262 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:04.869269 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:04.869643 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:04.870119 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:04.870132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:04.871174 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:04.871179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:04.872757 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:04.872792 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:04.873419 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:04.873426 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:04.873665 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:04.874295 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:04.874321 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:04.875969 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:04.877791 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:04.877796 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:04.877810 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:04.878057 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:04.878357 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:04.878361 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:04.878363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:04.878520 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:04.878981 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:04.878987 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:04.879308 (+ 321us) client_negotiation.cc:769] Sending connection context
0929 01:06:04.879508 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 01:06:04.879863 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":126,"threads_started":1}
I20260929 01:06:04.882206 17752 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43525
I20260929 01:06:04.882438 17752 heartbeater.cc:464] Registering TS with master...
I20260929 01:06:04.882995 17752 heartbeater.cc:511] Master 127.0.98.62:43525 requested a full tablet report, sending...
I20260929 01:06:04.884087 17427 ts_manager.cc:194] Registered new tserver with Master: c3dd4fe49cd2475495299b59666b4986 (127.0.98.2:43569)
I20260929 01:06:04.884981 17427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47105
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 01:06:04.895622 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43525
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:05.005137 17757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:05.005435 17757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:05.005508 17757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:05.009510 17757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:05.009600 17757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:05.009690 17757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:06:05.015394 17757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:05.016440 17757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:05.017434 17757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:05.026961 17765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:05.026957 17762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:05.026978 17763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:05.027343 17757 server_base.cc:1034] running on GCE node
I20260929 01:06:05.027737 17757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:05.028393 17757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:05.029556 17757 hybrid_clock.cc:648] HybridClock initialized: now 1790643965029539 us; error 27 us; skew 500 ppm
Sep 29 01:06:05 dist-test-slave-7zdl krb5kdc[17382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:05.032706 17757 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:06:05.033528 17757 webserver.cc:460] Webserver started at http://127.0.98.3:36537/ using document root <none> and password file <none>
I20260929 01:06:05.034044 17757 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:05.034101 17757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:05.034271 17757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:05.036235 17757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "64ca05964a4549cd87d59888e88aca2c"
format_stamp: "Formatted at 2026-09-29 01:06:05 on dist-test-slave-7zdl"
server_key: "e66a1d5b7ce2bbc54aad87062f731030"
server_key_iv: "a192ef2ca9224b44e389959a891e3527"
server_key_version: "encryptionkey@0"
I20260929 01:06:05.036731 17757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "64ca05964a4549cd87d59888e88aca2c"
format_stamp: "Formatted at 2026-09-29 01:06:05 on dist-test-slave-7zdl"
server_key: "e66a1d5b7ce2bbc54aad87062f731030"
server_key_iv: "a192ef2ca9224b44e389959a891e3527"
server_key_version: "encryptionkey@0"
I20260929 01:06:05.040232 17757 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:06:05.042420 17772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:05.043751 17757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:06:05.043874 17757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "64ca05964a4549cd87d59888e88aca2c"
format_stamp: "Formatted at 2026-09-29 01:06:05 on dist-test-slave-7zdl"
server_key: "e66a1d5b7ce2bbc54aad87062f731030"
server_key_iv: "a192ef2ca9224b44e389959a891e3527"
server_key_version: "encryptionkey@0"
I20260929 01:06:05.043967 17757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:05.058248 17757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:05.061398 17757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:05.061595 17757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:05.062152 17757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:06:05.063017 17757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:06:05.063063 17757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:05.063128 17757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:06:05.063167 17757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:05.071949 17757 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37379
I20260929 01:06:05.072011 17885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37379 every 8 connection(s)
I20260929 01:06:05.072818 17757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:06:05.081429 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 17757
I20260929 01:06:05.081534 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:06:05.081844 392 external_mini_cluster.cc:1428] Setting key cc40377156c891ef6087ad2c05593a1a
Sep 29 01:06:05 dist-test-slave-7zdl krb5kdc[17382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:06:05.086535 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:05.074889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40051 (local address 127.0.98.62:43525)
0929 01:06:05.075027 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:05.075039 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:05.075891 (+ 852us) server_negotiation.cc:407] Connection header received
0929 01:06:05.077170 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:05.077172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:05.077241 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:05.077324 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:05.078478 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.079030 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.080216 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.080424 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.080502 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:05.083170 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:05.083187 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:05.083190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:05.083219 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:05.084760 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:05.085259 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:05.085262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:05.085263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:05.085331 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:05.085627 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:05.085629 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:05.085630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:05.085801 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:05.085872 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:05.086331 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 01:06:05.086419 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 01:06:05.087397 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:05.075270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43525 (local address 127.0.98.3:40051)
0929 01:06:05.075732 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:05.075774 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:05.076888 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:05.077530 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:05.077537 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:05.077878 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:05.078338 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:05.078351 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.079158 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.079161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:05.080096 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:05.080105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.080551 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.080554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:05.080693 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:05.081307 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:05.081322 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:05.082972 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:05.084876 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:05.084880 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:05.084892 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:05.085174 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:05.085406 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:05.085409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:05.085410 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:05.085552 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:05.085885 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:05.085890 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:05.086135 (+ 245us) client_negotiation.cc:769] Sending connection context
0929 01:06:05.086288 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 01:06:05.086541 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":110,"threads_started":1}
I20260929 01:06:05.088486 17886 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43525
I20260929 01:06:05.088726 17886 heartbeater.cc:464] Registering TS with master...
I20260929 01:06:05.089257 17886 heartbeater.cc:511] Master 127.0.98.62:43525 requested a full tablet report, sending...
I20260929 01:06:05.090286 17427 ts_manager.cc:194] Registered new tserver with Master: 64ca05964a4549cd87d59888e88aca2c (127.0.98.3:37379)
I20260929 01:06:05.090912 17427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40051
I20260929 01:06:05.095775 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:06:05.105095 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:05.097682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36464 (local address 127.0.98.62:43525)
0929 01:06:05.097793 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:05.097797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:05.097957 (+ 160us) server_negotiation.cc:407] Connection header received
0929 01:06:05.098069 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:05.098072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:05.098117 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:05.098211 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:05.098941 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.099435 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.100288 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.100487 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.100560 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:05.102016 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:05.102019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:05.102021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:05.102040 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:05.103542 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:05.103961 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:05.103963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:05.103965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:05.104044 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:05.104271 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:05.104273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:05.104274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:05.104467 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:05.104572 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:05.104886 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 01:06:05.104982 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 01:06:05.107892 17427 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:06:05.110918 17427 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:06:05.125794 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:05.119936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43569 (local address 127.0.0.1:59752)
0929 01:06:05.120363 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:05.120376 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:05.120465 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:05.121095 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:05.121098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:05.121124 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:05.121257 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:05.121262 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.122564 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.122567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:05.123987 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:05.123997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.125219 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.125225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:05.125343 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:05.125403 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 01:06:05.125509 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 01:06:05.125626 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"spinlock_wait_cycles":768,"thread_start_us":97,"threads_started":1}
I20260929 01:06:05.126083 17897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:05.119155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33157 (local address 127.0.0.1:45460)
0929 01:06:05.119752 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:05.119805 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:05.119945 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:05.121042 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:05.121048 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:05.121070 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:05.121246 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:05.121254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.123260 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.123264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:05.124444 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:05.124457 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.125656 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.125660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:05.125762 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:05.125784 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 01:06:05.125856 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 01:06:05.125941 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":88,"spinlock_wait_cycles":4352,"thread_start_us":82,"threads_started":1}
I20260929 01:06:05.126127 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:05.120408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48446 (local address 127.0.98.3:37379)
0929 01:06:05.121400 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:05.121409 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:05.121425 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:06:05.121482 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:05.121485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:05.121602 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:05.121712 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:05.121960 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.123114 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.124490 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.125194 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.125297 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:05.125436 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:05.125863 (+ 427us) server_negotiation.cc:299] Negotiation successful
0929 01:06:05.125967 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":881,"thread_start_us":156,"threads_started":1}
I20260929 01:06:05.126381 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:05.120328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59752 (local address 127.0.98.2:43569)
0929 01:06:05.120700 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:05.120709 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:05.120726 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:06:05.120783 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:05.120787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:05.120926 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:05.121090 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:05.121380 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.122427 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.124110 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.125087 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.125201 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:05.125354 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:05.126064 (+ 710us) server_negotiation.cc:299] Negotiation successful
0929 01:06:05.126175 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":101,"threads_started":1}
I20260929 01:06:05.126551 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:05.119326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45460 (local address 127.0.98.1:33157)
0929 01:06:05.119755 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:05.119772 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:05.120097 (+ 325us) server_negotiation.cc:407] Connection header received
0929 01:06:05.120265 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:05.120270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:05.120885 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:05.121018 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:05.121843 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.123059 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.124579 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.125365 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.125463 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:05.125599 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:05.126300 (+ 701us) server_negotiation.cc:299] Negotiation successful
0929 01:06:05.126397 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":91,"threads_started":1}
I20260929 01:06:05.126320 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:05.119978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37379 (local address 127.0.0.1:48446)
0929 01:06:05.120451 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:05.120465 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:05.120554 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:05.121728 (+ 1174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:05.121731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:05.121749 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:05.121838 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:05.121843 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.123318 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.123320 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:05.124362 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:05.124372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.125498 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.125504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:05.125593 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:05.125614 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 01:06:05.125691 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 01:06:05.125745 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":65,"thread_start_us":107,"threads_started":1}
I20260929 01:06:05.129572 17552 tablet_service.cc:1467] Processing CreateTablet for tablet 7f0eff1ffaa54d38a06871462b395e2a (DEFAULT_TABLE table=test-table [id=b0442724fd9543c19dab4fff080a3e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:06:05.129594 17686 tablet_service.cc:1467] Processing CreateTablet for tablet 7f0eff1ffaa54d38a06871462b395e2a (DEFAULT_TABLE table=test-table [id=b0442724fd9543c19dab4fff080a3e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:06:05.129711 17820 tablet_service.cc:1467] Processing CreateTablet for tablet 7f0eff1ffaa54d38a06871462b395e2a (DEFAULT_TABLE table=test-table [id=b0442724fd9543c19dab4fff080a3e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:06:05.130687 17820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f0eff1ffaa54d38a06871462b395e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:05.130692 17552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f0eff1ffaa54d38a06871462b395e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:05.130712 17686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f0eff1ffaa54d38a06871462b395e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:05.136706 17903 tablet_bootstrap.cc:492] T 7f0eff1ffaa54d38a06871462b395e2a P 64ca05964a4549cd87d59888e88aca2c: Bootstrap starting.
I20260929 01:06:05.138314 17905 tablet_bootstrap.cc:492] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e: Bootstrap starting.
I20260929 01:06:05.138942 17904 tablet_bootstrap.cc:492] T 7f0eff1ffaa54d38a06871462b395e2a P c3dd4fe49cd2475495299b59666b4986: Bootstrap starting.
I20260929 01:06:05.139330 17903 tablet_bootstrap.cc:654] T 7f0eff1ffaa54d38a06871462b395e2a P 64ca05964a4549cd87d59888e88aca2c: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:05.140564 17905 tablet_bootstrap.cc:654] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:05.140851 17904 tablet_bootstrap.cc:654] T 7f0eff1ffaa54d38a06871462b395e2a P c3dd4fe49cd2475495299b59666b4986: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:05.141038 17903 log.cc:824] T 7f0eff1ffaa54d38a06871462b395e2a P 64ca05964a4549cd87d59888e88aca2c: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:05.141716 17905 log.cc:824] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:05.142015 17904 log.cc:824] T 7f0eff1ffaa54d38a06871462b395e2a P c3dd4fe49cd2475495299b59666b4986: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:05.143249 17903 tablet_bootstrap.cc:492] T 7f0eff1ffaa54d38a06871462b395e2a P 64ca05964a4549cd87d59888e88aca2c: No bootstrap required, opened a new log
I20260929 01:06:05.143440 17903 ts_tablet_manager.cc:1397] T 7f0eff1ffaa54d38a06871462b395e2a P 64ca05964a4549cd87d59888e88aca2c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 01:06:05.143517 17905 tablet_bootstrap.cc:492] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e: No bootstrap required, opened a new log
I20260929 01:06:05.143695 17905 ts_tablet_manager.cc:1397] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 01:06:05.143926 17904 tablet_bootstrap.cc:492] T 7f0eff1ffaa54d38a06871462b395e2a P c3dd4fe49cd2475495299b59666b4986: No bootstrap required, opened a new log
I20260929 01:06:05.144085 17904 ts_tablet_manager.cc:1397] T 7f0eff1ffaa54d38a06871462b395e2a P c3dd4fe49cd2475495299b59666b4986: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 01:06:05.146471 17905 raft_consensus.cc:348] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e6939f5074c7aa4d3b2d12d31871e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33157 } } peers { permanent_uuid: "c3dd4fe49cd2475495299b59666b4986" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43569 } } peers { permanent_uuid: "64ca05964a4549cd87d59888e88aca2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37379 } }
I20260929 01:06:05.146713 17905 raft_consensus.cc:374] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:05.146765 17905 raft_consensus.cc:729] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83e6939f5074c7aa4d3b2d12d31871e, State: Initialized, Role: FOLLOWER
I20260929 01:06:05.147310 17905 consensus_queue.cc:260] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e6939f5074c7aa4d3b2d12d31871e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33157 } } peers { permanent_uuid: "c3dd4fe49cd2475495299b59666b4986" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43569 } } peers { permanent_uuid: "64ca05964a4549cd87d59888e88aca2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37379 } }
I20260929 01:06:05.147578 17903 raft_consensus.cc:348] T 7f0eff1ffaa54d38a06871462b395e2a P 64ca05964a4549cd87d59888e88aca2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e6939f5074c7aa4d3b2d12d31871e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33157 } } peers { permanent_uuid: "c3dd4fe49cd2475495299b59666b4986" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43569 } } peers { permanent_uuid: "64ca05964a4549cd87d59888e88aca2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37379 } }
I20260929 01:06:05.147864 17903 raft_consensus.cc:374] T 7f0eff1ffaa54d38a06871462b395e2a P 64ca05964a4549cd87d59888e88aca2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:05.147926 17903 raft_consensus.cc:729] T 7f0eff1ffaa54d38a06871462b395e2a P 64ca05964a4549cd87d59888e88aca2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ca05964a4549cd87d59888e88aca2c, State: Initialized, Role: FOLLOWER
I20260929 01:06:05.147783 17904 raft_consensus.cc:348] T 7f0eff1ffaa54d38a06871462b395e2a P c3dd4fe49cd2475495299b59666b4986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e6939f5074c7aa4d3b2d12d31871e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33157 } } peers { permanent_uuid: "c3dd4fe49cd2475495299b59666b4986" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43569 } } peers { permanent_uuid: "64ca05964a4549cd87d59888e88aca2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37379 } }
I20260929 01:06:05.148032 17904 raft_consensus.cc:374] T 7f0eff1ffaa54d38a06871462b395e2a P c3dd4fe49cd2475495299b59666b4986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:05.148087 17904 raft_consensus.cc:729] T 7f0eff1ffaa54d38a06871462b395e2a P c3dd4fe49cd2475495299b59666b4986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3dd4fe49cd2475495299b59666b4986, State: Initialized, Role: FOLLOWER
I20260929 01:06:05.148393 17905 ts_tablet_manager.cc:1428] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 01:06:05.148348 17903 consensus_queue.cc:260] T 7f0eff1ffaa54d38a06871462b395e2a P 64ca05964a4549cd87d59888e88aca2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e6939f5074c7aa4d3b2d12d31871e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33157 } } peers { permanent_uuid: "c3dd4fe49cd2475495299b59666b4986" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43569 } } peers { permanent_uuid: "64ca05964a4549cd87d59888e88aca2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37379 } }
I20260929 01:06:05.148450 17904 consensus_queue.cc:260] T 7f0eff1ffaa54d38a06871462b395e2a P c3dd4fe49cd2475495299b59666b4986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e6939f5074c7aa4d3b2d12d31871e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33157 } } peers { permanent_uuid: "c3dd4fe49cd2475495299b59666b4986" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43569 } } peers { permanent_uuid: "64ca05964a4549cd87d59888e88aca2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37379 } }
I20260929 01:06:05.148780 17618 heartbeater.cc:503] Master 127.0.98.62:43525 was elected leader, sending a full tablet report...
I20260929 01:06:05.149300 17903 ts_tablet_manager.cc:1428] T 7f0eff1ffaa54d38a06871462b395e2a P 64ca05964a4549cd87d59888e88aca2c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 01:06:05.149333 17904 ts_tablet_manager.cc:1428] T 7f0eff1ffaa54d38a06871462b395e2a P c3dd4fe49cd2475495299b59666b4986: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 01:06:05.149637 17886 heartbeater.cc:503] Master 127.0.98.62:43525 was elected leader, sending a full tablet report...
I20260929 01:06:05.149866 17752 heartbeater.cc:503] Master 127.0.98.62:43525 was elected leader, sending a full tablet report...
I20260929 01:06:05.258821 17909 raft_consensus.cc:482] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:06:05.259027 17909 raft_consensus.cc:504] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e6939f5074c7aa4d3b2d12d31871e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33157 } } peers { permanent_uuid: "c3dd4fe49cd2475495299b59666b4986" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43569 } } peers { permanent_uuid: "64ca05964a4549cd87d59888e88aca2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37379 } }
I20260929 01:06:05.260295 17909 leader_election.cc:288] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3dd4fe49cd2475495299b59666b4986 (127.0.98.2:43569), 64ca05964a4549cd87d59888e88aca2c (127.0.98.3:37379)
I20260929 01:06:05.264159 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:05.260425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43569 (local address 127.0.98.1:35011)
0929 01:06:05.260754 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:05.260770 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:05.260934 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:05.261247 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:05.261252 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:05.261272 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:05.261428 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:05.261434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.262163 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.262169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:05.263056 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:05.263065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.263730 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.263733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:05.263826 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:05.263871 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 01:06:05.263929 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 01:06:05.263979 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":15,"thread_start_us":116,"threads_started":1}
I20260929 01:06:05.264194 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:05.260532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35011 (local address 127.0.98.2:43569)
0929 01:06:05.260696 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:05.260703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:05.260883 (+ 180us) server_negotiation.cc:407] Connection header received
0929 01:06:05.261070 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:05.261076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:05.261143 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:05.261207 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:05.261547 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.262049 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.263187 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.263607 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.263726 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:05.263790 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:05.263972 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 01:06:05.264055 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 01:06:05.264271 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:05.260795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52091 (local address 127.0.98.3:37379)
0929 01:06:05.260972 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:05.260978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:05.260997 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:06:05.261135 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:05.261138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:05.261179 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:05.261276 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:05.261573 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.262127 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.263219 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.263609 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.263726 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:05.263786 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:05.264074 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 01:06:05.264135 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 01:06:05.264232 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:05.260530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37379 (local address 127.0.98.1:52091)
0929 01:06:05.260892 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:05.260907 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:05.261030 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:05.261310 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:05.261314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:05.261331 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:05.261437 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:05.261442 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.262237 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.262240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:05.263081 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:05.263089 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.263731 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.263734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:05.263933 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:05.263960 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 01:06:05.264036 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 01:06:05.264095 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":46,"thread_start_us":113,"threads_started":1}
I20260929 01:06:05.264966 17840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0eff1ffaa54d38a06871462b395e2a" candidate_uuid: "f83e6939f5074c7aa4d3b2d12d31871e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ca05964a4549cd87d59888e88aca2c" is_pre_election: true
I20260929 01:06:05.264971 17706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0eff1ffaa54d38a06871462b395e2a" candidate_uuid: "f83e6939f5074c7aa4d3b2d12d31871e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3dd4fe49cd2475495299b59666b4986" is_pre_election: true
I20260929 01:06:05.265257 17840 raft_consensus.cc:2454] T 7f0eff1ffaa54d38a06871462b395e2a P 64ca05964a4549cd87d59888e88aca2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83e6939f5074c7aa4d3b2d12d31871e in term 0.
I20260929 01:06:05.265280 17706 raft_consensus.cc:2454] T 7f0eff1ffaa54d38a06871462b395e2a P c3dd4fe49cd2475495299b59666b4986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83e6939f5074c7aa4d3b2d12d31871e in term 0.
I20260929 01:06:05.265728 17507 leader_election.cc:302] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64ca05964a4549cd87d59888e88aca2c, f83e6939f5074c7aa4d3b2d12d31871e; no voters:
I20260929 01:06:05.266007 17909 raft_consensus.cc:2781] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:06:05.266096 17909 raft_consensus.cc:482] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:06:05.266139 17909 raft_consensus.cc:3037] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:05.267387 17909 raft_consensus.cc:504] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e6939f5074c7aa4d3b2d12d31871e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33157 } } peers { permanent_uuid: "c3dd4fe49cd2475495299b59666b4986" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43569 } } peers { permanent_uuid: "64ca05964a4549cd87d59888e88aca2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37379 } }
I20260929 01:06:05.267763 17909 leader_election.cc:288] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e [CANDIDATE]: Term 1 election: Requested vote from peers c3dd4fe49cd2475495299b59666b4986 (127.0.98.2:43569), 64ca05964a4549cd87d59888e88aca2c (127.0.98.3:37379)
I20260929 01:06:05.268074 17706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0eff1ffaa54d38a06871462b395e2a" candidate_uuid: "f83e6939f5074c7aa4d3b2d12d31871e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3dd4fe49cd2475495299b59666b4986"
I20260929 01:06:05.268114 17840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0eff1ffaa54d38a06871462b395e2a" candidate_uuid: "f83e6939f5074c7aa4d3b2d12d31871e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ca05964a4549cd87d59888e88aca2c"
I20260929 01:06:05.268203 17840 raft_consensus.cc:3037] T 7f0eff1ffaa54d38a06871462b395e2a P 64ca05964a4549cd87d59888e88aca2c [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:05.268224 17706 raft_consensus.cc:3037] T 7f0eff1ffaa54d38a06871462b395e2a P c3dd4fe49cd2475495299b59666b4986 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:05.269176 17840 raft_consensus.cc:2454] T 7f0eff1ffaa54d38a06871462b395e2a P 64ca05964a4549cd87d59888e88aca2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83e6939f5074c7aa4d3b2d12d31871e in term 1.
I20260929 01:06:05.269330 17706 raft_consensus.cc:2454] T 7f0eff1ffaa54d38a06871462b395e2a P c3dd4fe49cd2475495299b59666b4986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83e6939f5074c7aa4d3b2d12d31871e in term 1.
I20260929 01:06:05.269481 17507 leader_election.cc:302] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64ca05964a4549cd87d59888e88aca2c, f83e6939f5074c7aa4d3b2d12d31871e; no voters:
I20260929 01:06:05.269650 17909 raft_consensus.cc:2781] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:05.269891 17909 raft_consensus.cc:686] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e [term 1 LEADER]: Becoming Leader. State: Replica: f83e6939f5074c7aa4d3b2d12d31871e, State: Running, Role: LEADER
I20260929 01:06:05.270213 17909 consensus_queue.cc:237] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e6939f5074c7aa4d3b2d12d31871e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33157 } } peers { permanent_uuid: "c3dd4fe49cd2475495299b59666b4986" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43569 } } peers { permanent_uuid: "64ca05964a4549cd87d59888e88aca2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37379 } }
I20260929 01:06:05.273444 17427 catalog_manager.cc:5495] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e reported cstate change: term changed from 0 to 1, leader changed from <none> to f83e6939f5074c7aa4d3b2d12d31871e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f83e6939f5074c7aa4d3b2d12d31871e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e6939f5074c7aa4d3b2d12d31871e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3dd4fe49cd2475495299b59666b4986" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64ca05964a4549cd87d59888e88aca2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37379 } health_report { overall_health: UNKNOWN } } }
I20260929 01:06:05.312144 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:05.308689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45474 (local address 127.0.98.1:33157)
0929 01:06:05.308808 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:05.308812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:05.308841 (+ 29us) server_negotiation.cc:407] Connection header received
0929 01:06:05.308930 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:05.308931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:05.308982 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:05.309070 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:05.309344 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.309856 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.310739 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.310979 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.311053 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:05.311361 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:05.311778 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:05.311866 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:05.311980 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 01:06:05.312026 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 01:06:05.318104 17706 raft_consensus.cc:1261] T 7f0eff1ffaa54d38a06871462b395e2a P c3dd4fe49cd2475495299b59666b4986 [term 1 FOLLOWER]: Refusing update from remote peer f83e6939f5074c7aa4d3b2d12d31871e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:06:05.318130 17840 raft_consensus.cc:1261] T 7f0eff1ffaa54d38a06871462b395e2a P 64ca05964a4549cd87d59888e88aca2c [term 1 FOLLOWER]: Refusing update from remote peer f83e6939f5074c7aa4d3b2d12d31871e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:06:05.318840 17909 consensus_queue.cc:1035] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ca05964a4549cd87d59888e88aca2c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:06:05.319053 17914 consensus_queue.cc:1035] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3dd4fe49cd2475495299b59666b4986" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 01:06:05.323738 17887 tablet.cc:2369] T 7f0eff1ffaa54d38a06871462b395e2a P 64ca05964a4549cd87d59888e88aca2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:06:05.326498 17919 mvcc.cc:205] Tried to move back new op lower bound from 7334477681936039936 to 7334477681749835776. Current Snapshot: MvccSnapshot[applied={T|T < 7334477681936039936}]
I20260929 01:06:05.327454 17918 mvcc.cc:205] Tried to move back new op lower bound from 7334477681936039936 to 7334477681749835776. Current Snapshot: MvccSnapshot[applied={T|T < 7334477681936039936}]
I20260929 01:06:05.335314 17425 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:06:05.335469 17425 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:06:05.337942 17425 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37e84fdbfe4c419094bc8f6d7c9f9c68: Sending DeleteTablet for 3 replicas of tablet 7f0eff1ffaa54d38a06871462b395e2a
I20260929 01:06:05.338864 17820 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f0eff1ffaa54d38a06871462b395e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:06:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48446
I20260929 01:06:05.338887 17552 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f0eff1ffaa54d38a06871462b395e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:06:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45460
I20260929 01:06:05.338984 17686 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f0eff1ffaa54d38a06871462b395e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:06:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59752
I20260929 01:06:05.339459 17926 tablet_replica.cc:330] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e: stopping tablet replica
I20260929 01:06:05.339707 17928 tablet_replica.cc:330] T 7f0eff1ffaa54d38a06871462b395e2a P c3dd4fe49cd2475495299b59666b4986: stopping tablet replica
I20260929 01:06:05.339800 17926 raft_consensus.cc:2229] T 7f0eff1ffaa54d38a06871462b395e2a P f83e6939f5074c7aa4d3b2d12d31871e [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:06:05.340042 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 17489
I20260929 01:06:05.341049 17928 raft_consensus.cc:2229] T 7f0eff1ffaa54d38a06871462b395e2a P c3dd4fe49cd2475495299b59666b4986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:06:05.341440 17928 raft_consensus.cc:2258] T 7f0eff1ffaa54d38a06871462b395e2a P c3dd4fe49cd2475495299b59666b4986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:06:05.341814 17927 tablet_replica.cc:330] T 7f0eff1ffaa54d38a06871462b395e2a P 64ca05964a4549cd87d59888e88aca2c: stopping tablet replica
I20260929 01:06:05.342088 17927 raft_consensus.cc:2229] T 7f0eff1ffaa54d38a06871462b395e2a P 64ca05964a4549cd87d59888e88aca2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:06:05.342381 17927 raft_consensus.cc:2258] T 7f0eff1ffaa54d38a06871462b395e2a P 64ca05964a4549cd87d59888e88aca2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:06:05.342988 17928 ts_tablet_manager.cc:1905] T 7f0eff1ffaa54d38a06871462b395e2a P c3dd4fe49cd2475495299b59666b4986: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:06:05.343927 17927 ts_tablet_manager.cc:1905] T 7f0eff1ffaa54d38a06871462b395e2a P 64ca05964a4549cd87d59888e88aca2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:06:05.345317 17928 ts_tablet_manager.cc:1918] T 7f0eff1ffaa54d38a06871462b395e2a P c3dd4fe49cd2475495299b59666b4986: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:06:05.345427 17928 log.cc:1196] T 7f0eff1ffaa54d38a06871462b395e2a P c3dd4fe49cd2475495299b59666b4986: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-1/wal/wals/7f0eff1ffaa54d38a06871462b395e2a
I20260929 01:06:05.345712 17928 ts_tablet_manager.cc:1939] T 7f0eff1ffaa54d38a06871462b395e2a P c3dd4fe49cd2475495299b59666b4986: Deleting consensus metadata
I20260929 01:06:05.347031 17414 catalog_manager.cc:4841] TS c3dd4fe49cd2475495299b59666b4986 (127.0.98.2:43569): tablet 7f0eff1ffaa54d38a06871462b395e2a (table test-table [id=b0442724fd9543c19dab4fff080a3e0d]) successfully deleted
I20260929 01:06:05.347493 17927 ts_tablet_manager.cc:1918] T 7f0eff1ffaa54d38a06871462b395e2a P 64ca05964a4549cd87d59888e88aca2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:06:05.347597 17927 log.cc:1196] T 7f0eff1ffaa54d38a06871462b395e2a P 64ca05964a4549cd87d59888e88aca2c: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/7f0eff1ffaa54d38a06871462b395e2a
I20260929 01:06:05.347999 17927 ts_tablet_manager.cc:1939] T 7f0eff1ffaa54d38a06871462b395e2a P 64ca05964a4549cd87d59888e88aca2c: Deleting consensus metadata
W20260929 01:06:05.348752 17414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33157 (error 108)
I20260929 01:06:05.349413 17413 catalog_manager.cc:4841] TS 64ca05964a4549cd87d59888e88aca2c (127.0.98.3:37379): tablet 7f0eff1ffaa54d38a06871462b395e2a (table test-table [id=b0442724fd9543c19dab4fff080a3e0d]) successfully deleted
I20260929 01:06:05.349715 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:05.349452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33157 (local address 127.0.0.1:45490)
0929 01:06:05.349576 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:05.349642 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 01:06:05.350016 17414 catalog_manager.cc:4584] TS f83e6939f5074c7aa4d3b2d12d31871e (127.0.98.1:33157): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f0eff1ffaa54d38a06871462b395e2a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33157: connect: Connection refused (error 111)
I20260929 01:06:05.350322 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 17623
I20260929 01:06:05.357082 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 17757
I20260929 01:06:05.362648 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 17395
2026-09-29T01:06:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2346 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T01:06:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:05Z Disabled control of system clock
I20260929 01:06:05.392452 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37983
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 01:06:05.503141 17933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:05.503391 17933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:05.503425 17933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:05.507396 17933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:05.507448 17933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:05.507464 17933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:05.507480 17933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:05.507495 17933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:05.512058 17933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37983
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:05.513029 17933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:05.514400 17933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:05.523082 17938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:05.523106 17941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:05.523299 17933 server_base.cc:1034] running on GCE node
W20260929 01:06:05.523125 17939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:05.523833 17933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:05.524945 17933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:05.526134 17933 hybrid_clock.cc:648] HybridClock initialized: now 1790643965526118 us; error 28 us; skew 500 ppm
I20260929 01:06:05.528020 17933 webserver.cc:460] Webserver started at http://127.0.98.62:39669/ using document root <none> and password file <none>
I20260929 01:06:05.528612 17933 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:05.528697 17933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:05.528900 17933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:05.530957 17933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "8d36e24df26247d6b9b5dc77545ad380"
format_stamp: "Formatted at 2026-09-29 01:06:05 on dist-test-slave-7zdl"
server_key: "74871aa0feeedecdcaaa65f8caf9f349"
server_key_iv: "ecd13c2d92fd172875da1450af44778a"
server_key_version: "encryptionkey@0"
I20260929 01:06:05.531522 17933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "8d36e24df26247d6b9b5dc77545ad380"
format_stamp: "Formatted at 2026-09-29 01:06:05 on dist-test-slave-7zdl"
server_key: "74871aa0feeedecdcaaa65f8caf9f349"
server_key_iv: "ecd13c2d92fd172875da1450af44778a"
server_key_version: "encryptionkey@0"
I20260929 01:06:05.535166 17933 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 01:06:05.537569 17947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:05.538898 17933 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:06:05.539029 17933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "8d36e24df26247d6b9b5dc77545ad380"
format_stamp: "Formatted at 2026-09-29 01:06:05 on dist-test-slave-7zdl"
server_key: "74871aa0feeedecdcaaa65f8caf9f349"
server_key_iv: "ecd13c2d92fd172875da1450af44778a"
server_key_version: "encryptionkey@0"
I20260929 01:06:05.539140 17933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:05.553719 17933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:05.554388 17933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:05.554575 17933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:05.561390 17933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37983
I20260929 01:06:05.561403 17999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37983 every 8 connection(s)
I20260929 01:06:05.562309 17933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:05.565546 18000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:05.567965 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 17933
I20260929 01:06:05.568120 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:05.568409 392 external_mini_cluster.cc:1428] Setting key 5ead308ad4c4f4e7e0804fd2e0d3d963
I20260929 01:06:05.572572 18000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d36e24df26247d6b9b5dc77545ad380: Bootstrap starting.
I20260929 01:06:05.575408 18000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d36e24df26247d6b9b5dc77545ad380: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:05.576232 18000 log.cc:824] T 00000000000000000000000000000000 P 8d36e24df26247d6b9b5dc77545ad380: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:05.578750 18000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d36e24df26247d6b9b5dc77545ad380: No bootstrap required, opened a new log
I20260929 01:06:05.579393 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:05.569888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52138 (local address 127.0.98.62:37983)
0929 01:06:05.570612 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:05.570632 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:05.570679 (+ 47us) server_negotiation.cc:407] Connection header received
0929 01:06:05.571607 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:05.571634 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:05.572157 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:05.572551 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:05.573026 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.574624 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.575635 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.576112 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.576292 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:05.577302 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:05.577311 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:05.577330 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:05.577816 (+ 486us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:05.577887 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:05.577916 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:05.578027 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:05.578130 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 01:06:05.578385 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":113,"threads_started":1}
I20260929 01:06:05.582679 18000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d36e24df26247d6b9b5dc77545ad380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d36e24df26247d6b9b5dc77545ad380" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37983 } }
I20260929 01:06:05.582955 18000 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d36e24df26247d6b9b5dc77545ad380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:05.583009 18000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d36e24df26247d6b9b5dc77545ad380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d36e24df26247d6b9b5dc77545ad380, State: Initialized, Role: FOLLOWER
I20260929 01:06:05.583397 18000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d36e24df26247d6b9b5dc77545ad380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d36e24df26247d6b9b5dc77545ad380" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37983 } }
I20260929 01:06:05.583525 18000 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d36e24df26247d6b9b5dc77545ad380 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:05.583570 18000 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d36e24df26247d6b9b5dc77545ad380 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:05.583638 18000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d36e24df26247d6b9b5dc77545ad380 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:05.584621 18000 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d36e24df26247d6b9b5dc77545ad380 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d36e24df26247d6b9b5dc77545ad380" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37983 } }
I20260929 01:06:05.584903 18000 leader_election.cc:302] T 00000000000000000000000000000000 P 8d36e24df26247d6b9b5dc77545ad380 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d36e24df26247d6b9b5dc77545ad380; no voters:
I20260929 01:06:05.585180 18000 leader_election.cc:288] T 00000000000000000000000000000000 P 8d36e24df26247d6b9b5dc77545ad380 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:05.585250 18005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d36e24df26247d6b9b5dc77545ad380 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:05.585503 18005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d36e24df26247d6b9b5dc77545ad380 [term 1 LEADER]: Becoming Leader. State: Replica: 8d36e24df26247d6b9b5dc77545ad380, State: Running, Role: LEADER
I20260929 01:06:05.585800 18005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d36e24df26247d6b9b5dc77545ad380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d36e24df26247d6b9b5dc77545ad380" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37983 } }
I20260929 01:06:05.586359 18000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d36e24df26247d6b9b5dc77545ad380 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:05.587540 18007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d36e24df26247d6b9b5dc77545ad380 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d36e24df26247d6b9b5dc77545ad380. Latest consensus state: current_term: 1 leader_uuid: "8d36e24df26247d6b9b5dc77545ad380" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d36e24df26247d6b9b5dc77545ad380" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37983 } } }
I20260929 01:06:05.587670 18007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d36e24df26247d6b9b5dc77545ad380 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:05.587884 18006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d36e24df26247d6b9b5dc77545ad380 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d36e24df26247d6b9b5dc77545ad380" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d36e24df26247d6b9b5dc77545ad380" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37983 } } }
I20260929 01:06:05.587976 18006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d36e24df26247d6b9b5dc77545ad380 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:05.588502 18014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:05.591184 18014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:05.596746 18014 catalog_manager.cc:1347] Generated new cluster ID: b1793d5cd1ec454ab8641642478b0adb
I20260929 01:06:05.596817 18014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:05.610037 18014 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:05.611043 18014 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:05.619045 18014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d36e24df26247d6b9b5dc77545ad380: Generated new TSK 0
I20260929 01:06:05.619812 18014 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:05.624305 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37983
--builtin_ntp_servers=127.0.98.20:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 01:06:05.751982 18024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:05.752254 18024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:05.752331 18024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:05.756366 18024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:05.756412 18024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:05.756466 18024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:06:05.761193 18024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:05.762241 18024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:05.763681 18024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:05.773923 18030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:05.773933 18029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:05.773916 18032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:05.773980 18024 server_base.cc:1034] running on GCE node
I20260929 01:06:05.774471 18024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:05.775154 18024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:05.776337 18024 hybrid_clock.cc:648] HybridClock initialized: now 1790643965776327 us; error 29 us; skew 500 ppm
I20260929 01:06:05.778378 18024 webserver.cc:460] Webserver started at http://127.0.98.1:34737/ using document root <none> and password file <none>
I20260929 01:06:05.779168 18024 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:05.779263 18024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:05.779498 18024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:05.782169 18024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "b80dba40734745db85ab077810cabd96"
format_stamp: "Formatted at 2026-09-29 01:06:05 on dist-test-slave-7zdl"
server_key: "19f4be8978e6945695eae2cb6376f0cc"
server_key_iv: "616e0d72403099afe93219d0213ed696"
server_key_version: "encryptionkey@0"
I20260929 01:06:05.782720 18024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b80dba40734745db85ab077810cabd96"
format_stamp: "Formatted at 2026-09-29 01:06:05 on dist-test-slave-7zdl"
server_key: "19f4be8978e6945695eae2cb6376f0cc"
server_key_iv: "616e0d72403099afe93219d0213ed696"
server_key_version: "encryptionkey@0"
I20260929 01:06:05.786403 18024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:06:05.789014 18038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:05.790223 18024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 01:06:05.790316 18024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "b80dba40734745db85ab077810cabd96"
format_stamp: "Formatted at 2026-09-29 01:06:05 on dist-test-slave-7zdl"
server_key: "19f4be8978e6945695eae2cb6376f0cc"
server_key_iv: "616e0d72403099afe93219d0213ed696"
server_key_version: "encryptionkey@0"
I20260929 01:06:05.790400 18024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:05.809036 18024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:05.809654 18024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:05.809798 18024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:05.810333 18024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:06:05.811228 18024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:06:05.811273 18024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:05.811313 18024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:06:05.811334 18024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:05.819986 18024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44877
I20260929 01:06:05.820139 18151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44877 every 8 connection(s)
I20260929 01:06:05.820940 18024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:06:05.821765 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 18024
I20260929 01:06:05.821882 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:06:05.822175 392 external_mini_cluster.cc:1428] Setting key 33de94a352ccbe7cbfc0c8e1495cdae6
I20260929 01:06:05.824163 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37983
--builtin_ntp_servers=127.0.98.20:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 01:06:05.832767 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:05.822933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60701 (local address 127.0.98.62:37983)
0929 01:06:05.823066 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:05.823070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:05.825773 (+ 2703us) server_negotiation.cc:407] Connection header received
0929 01:06:05.827193 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:05.827199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:05.827264 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:05.827367 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:05.828290 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.829255 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.830465 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.830917 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.831047 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:05.832176 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:05.832182 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:05.832188 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:05.832206 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:05.832234 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:05.832248 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:05.832333 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:05.832484 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 01:06:05.832559 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:06:05.833485 18154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:05.823382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37983 (local address 127.0.98.1:60701)
0929 01:06:05.825636 (+ 2254us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:05.825674 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:05.826981 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:05.827555 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:05.827562 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:05.827646 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:05.828149 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:05.828161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.829411 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.829414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:05.830341 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:05.830350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:05.831062 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:05.831065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:05.831238 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:05.831907 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:06:05.831909 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:05.831949 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:06:05.832009 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:05.832346 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:05.832353 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:05.832357 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 01:06:05.832448 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 01:06:05.832658 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1879,"thread_start_us":148,"threads_started":1}
I20260929 01:06:05.834748 18152 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37983
I20260929 01:06:05.835022 18152 heartbeater.cc:464] Registering TS with master...
I20260929 01:06:05.835516 18152 heartbeater.cc:511] Master 127.0.98.62:37983 requested a full tablet report, sending...
I20260929 01:06:05.837118 17964 ts_manager.cc:194] Registered new tserver with Master: b80dba40734745db85ab077810cabd96 (127.0.98.1:44877)
I20260929 01:06:05.838658 17964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:60701
W20260929 01:06:05.938907 18155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:05.939164 18155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:05.939204 18155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:05.943244 18155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:05.943298 18155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:05.943398 18155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:06:05.948153 18155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:05.949534 18155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:05.950994 18155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:05.960465 18160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:05.960484 18161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:05.960480 18163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:05.960692 18155 server_base.cc:1034] running on GCE node
I20260929 01:06:05.961141 18155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:05.961861 18155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:05.963048 18155 hybrid_clock.cc:648] HybridClock initialized: now 1790643965963024 us; error 41 us; skew 500 ppm
I20260929 01:06:05.965129 18155 webserver.cc:460] Webserver started at http://127.0.98.2:34445/ using document root <none> and password file <none>
I20260929 01:06:05.965864 18155 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:05.965953 18155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:05.966207 18155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:05.968453 18155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "b28b86860c1c43d29b36e297cd280f8a"
format_stamp: "Formatted at 2026-09-29 01:06:05 on dist-test-slave-7zdl"
server_key: "0f1cb93f7f72398a71dcffb15aa15208"
server_key_iv: "f02b6a5af94e112140caad14dee87357"
server_key_version: "encryptionkey@0"
I20260929 01:06:05.969151 18155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b28b86860c1c43d29b36e297cd280f8a"
format_stamp: "Formatted at 2026-09-29 01:06:05 on dist-test-slave-7zdl"
server_key: "0f1cb93f7f72398a71dcffb15aa15208"
server_key_iv: "f02b6a5af94e112140caad14dee87357"
server_key_version: "encryptionkey@0"
I20260929 01:06:05.973644 18155 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 01:06:05.976439 18169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:05.977818 18155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:06:05.977938 18155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "b28b86860c1c43d29b36e297cd280f8a"
format_stamp: "Formatted at 2026-09-29 01:06:05 on dist-test-slave-7zdl"
server_key: "0f1cb93f7f72398a71dcffb15aa15208"
server_key_iv: "f02b6a5af94e112140caad14dee87357"
server_key_version: "encryptionkey@0"
I20260929 01:06:05.978041 18155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:06.010885 18155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:06.011631 18155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:06.011833 18155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:06.012444 18155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:06:06.013427 18155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:06:06.013494 18155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:06.013576 18155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:06:06.013615 18155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:06.023433 18155 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32937
I20260929 01:06:06.023494 18282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32937 every 8 connection(s)
I20260929 01:06:06.024394 18155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:06:06.034202 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:06.026339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35397 (local address 127.0.98.62:37983)
0929 01:06:06.026507 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:06.026511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:06.027647 (+ 1136us) server_negotiation.cc:407] Connection header received
0929 01:06:06.029124 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:06.029128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:06.029192 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:06.029318 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:06.030292 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.030931 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.032148 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.032394 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.032519 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:06.033593 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:06.033598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:06.033601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:06.033623 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:06.033646 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:06.033655 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:06.033769 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:06.033938 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 01:06:06.034012 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:06:06.034436 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 18155
I20260929 01:06:06.034581 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:06:06.034937 392 external_mini_cluster.cc:1428] Setting key 25369315555813a05bf6d59b708b7822
I20260929 01:06:06.035243 18285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:06.026713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37983 (local address 127.0.98.2:35397)
0929 01:06:06.027468 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:06.027527 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:06.028864 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:06.029497 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:06.029504 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:06.029588 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:06.030133 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:06.030148 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.031072 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.031082 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:06.032011 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:06.032021 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.032557 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.032562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:06.032743 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:06.033305 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:06:06.033307 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:06.033348 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:06:06.033409 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:06.033769 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:06.033784 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:06.033793 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 01:06:06.033938 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 01:06:06.034208 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":127,"threads_started":1}
I20260929 01:06:06.036456 18283 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37983
I20260929 01:06:06.036691 18283 heartbeater.cc:464] Registering TS with master...
I20260929 01:06:06.036994 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37983
--builtin_ntp_servers=127.0.98.20:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 01:06:06.037233 18283 heartbeater.cc:511] Master 127.0.98.62:37983 requested a full tablet report, sending...
I20260929 01:06:06.038265 17964 ts_manager.cc:194] Registered new tserver with Master: b28b86860c1c43d29b36e297cd280f8a (127.0.98.2:32937)
I20260929 01:06:06.038965 17964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:35397
W20260929 01:06:06.146318 18286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:06.146589 18286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:06.146657 18286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:06.150648 18286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:06.150729 18286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:06.150861 18286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:06:06.155556 18286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:06.156733 18286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:06.158067 18286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:06.167240 18292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:06.167336 18286 server_base.cc:1034] running on GCE node
W20260929 01:06:06.167228 18291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:06.167258 18294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:06.167893 18286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:06.168517 18286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:06.169699 18286 hybrid_clock.cc:648] HybridClock initialized: now 1790643966169685 us; error 30 us; skew 500 ppm
I20260929 01:06:06.171543 18286 webserver.cc:460] Webserver started at http://127.0.98.3:46461/ using document root <none> and password file <none>
I20260929 01:06:06.172098 18286 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:06.172197 18286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:06.172432 18286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:06.174508 18286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "f34cd0eb2c7d493fb3b3d0b3782ad1a6"
format_stamp: "Formatted at 2026-09-29 01:06:06 on dist-test-slave-7zdl"
server_key: "751267c404af0de5371401ff6ac8300c"
server_key_iv: "e46c18b3fee0b281638b9ae272e7e089"
server_key_version: "encryptionkey@0"
I20260929 01:06:06.175077 18286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f34cd0eb2c7d493fb3b3d0b3782ad1a6"
format_stamp: "Formatted at 2026-09-29 01:06:06 on dist-test-slave-7zdl"
server_key: "751267c404af0de5371401ff6ac8300c"
server_key_iv: "e46c18b3fee0b281638b9ae272e7e089"
server_key_version: "encryptionkey@0"
I20260929 01:06:06.178623 18286 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:06:06.181018 18300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:06.182394 18286 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:06:06.182539 18286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "f34cd0eb2c7d493fb3b3d0b3782ad1a6"
format_stamp: "Formatted at 2026-09-29 01:06:06 on dist-test-slave-7zdl"
server_key: "751267c404af0de5371401ff6ac8300c"
server_key_iv: "e46c18b3fee0b281638b9ae272e7e089"
server_key_version: "encryptionkey@0"
I20260929 01:06:06.182658 18286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:06.207602 18286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:06.208366 18286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:06.208595 18286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:06.209250 18286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:06:06.210251 18286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:06:06.210330 18286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:06.210407 18286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:06:06.210454 18286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:06.220377 18286 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42671
I20260929 01:06:06.220402 18413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42671 every 8 connection(s)
I20260929 01:06:06.221334 18286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:06:06.223927 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 18286
I20260929 01:06:06.224057 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:06:06.224366 392 external_mini_cluster.cc:1428] Setting key 5f384dee2e8527cf1d3e2bd540e21a26
I20260929 01:06:06.230948 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:06.223293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58049 (local address 127.0.98.62:37983)
0929 01:06:06.223433 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:06.223436 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:06.224747 (+ 1311us) server_negotiation.cc:407] Connection header received
0929 01:06:06.226316 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:06.226319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:06.226372 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:06.226450 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:06.227405 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.227941 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.229086 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.229322 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.229438 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:06.230442 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:06.230446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:06.230449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:06.230467 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:06.230486 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:06.230493 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:06.230571 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:06.230724 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 01:06:06.230782 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 01:06:06.231788 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:06.223764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37983 (local address 127.0.98.3:58049)
0929 01:06:06.224586 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:06.224626 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:06.226114 (+ 1488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:06.226648 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:06.226663 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:06.226748 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:06.227274 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:06.227286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.228073 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.228077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:06.228961 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:06.228970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.229470 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.229475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:06.229643 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:06.230199 (+ 556us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:06:06.230201 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:06.230240 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:06:06.230298 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:06.230584 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:06.230593 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:06.230605 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 01:06:06.230723 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 01:06:06.231025 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":180,"threads_started":1}
I20260929 01:06:06.232831 18414 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37983
I20260929 01:06:06.233117 18414 heartbeater.cc:464] Registering TS with master...
I20260929 01:06:06.233620 18414 heartbeater.cc:511] Master 127.0.98.62:37983 requested a full tablet report, sending...
I20260929 01:06:06.234687 17964 ts_manager.cc:194] Registered new tserver with Master: f34cd0eb2c7d493fb3b3d0b3782ad1a6 (127.0.98.3:42671)
I20260929 01:06:06.235328 17964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:58049
I20260929 01:06:06.238225 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:06:06.244400 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:06.240175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52154 (local address 127.0.98.62:37983)
0929 01:06:06.240295 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:06.240298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:06.240544 (+ 246us) server_negotiation.cc:407] Connection header received
0929 01:06:06.240644 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:06.240647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:06.240694 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:06.240759 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:06.241102 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.241583 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.243036 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.243260 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.243335 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:06.244077 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:06.244080 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:06.244082 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:06.244092 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:06.244108 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:06.244115 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:06.244170 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:06.244256 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 01:06:06.244300 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 01:06:06.246980 17964 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:06:06.249859 17964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:06:06.263881 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:06.257963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32937 (local address 127.0.0.1:60984)
0929 01:06:06.258440 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:06.258485 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:06.258609 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:06.259838 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:06.259842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:06.259860 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:06.260002 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:06.260007 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.261083 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.261086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:06.262330 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:06.262340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.263323 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.263326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:06.263431 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:06.263466 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 01:06:06.263599 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 01:06:06.263718 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":34,"thread_start_us":86,"threads_started":1}
I20260929 01:06:06.264106 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:06.258988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57720 (local address 127.0.98.3:42671)
0929 01:06:06.259329 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:06.259342 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:06.259820 (+ 478us) server_negotiation.cc:407] Connection header received
0929 01:06:06.259893 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:06.259899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:06.260063 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:06.260189 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:06.260354 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.261414 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.262640 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.263471 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.263575 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:06.263776 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:06.263827 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 01:06:06.263923 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":137,"threads_started":1}
I20260929 01:06:06.263932 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:06.258817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42671 (local address 127.0.0.1:57720)
0929 01:06:06.259622 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:06.259636 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:06.259735 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:06.260167 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:06.260172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:06.260191 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:06.260266 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:06.260271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.261531 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.261534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:06.262512 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:06.262521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.263601 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.263606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:06.263684 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:06.263704 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 01:06:06.263761 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 01:06:06.263815 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":1,"spinlock_wait_cycles":384,"thread_start_us":88,"threads_started":1}
I20260929 01:06:06.264497 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:06.258119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60984 (local address 127.0.98.2:32937)
0929 01:06:06.259187 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:06.259203 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:06.259230 (+ 27us) server_negotiation.cc:407] Connection header received
0929 01:06:06.259308 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:06.259314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:06.259527 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:06.259678 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:06.260123 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.260951 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.262468 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.263195 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.263322 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:06.263525 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:06.264199 (+ 674us) server_negotiation.cc:299] Negotiation successful
0929 01:06:06.264314 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":943,"thread_start_us":106,"threads_started":1}
I20260929 01:06:06.264516 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:06.258150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44877 (local address 127.0.0.1:45512)
0929 01:06:06.258777 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:06.258821 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:06.258908 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:06.259699 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:06.259705 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:06.259736 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:06.259906 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:06.259912 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.261483 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.261486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:06.262676 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:06.262685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.263726 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.263730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:06.263818 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:06.263842 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 01:06:06.263891 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 01:06:06.263955 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":44,"spinlock_wait_cycles":4736,"thread_start_us":102,"threads_started":1}
I20260929 01:06:06.264559 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:06.258651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45512 (local address 127.0.98.1:44877)
0929 01:06:06.259198 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:06.259210 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:06.259232 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:06:06.259308 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:06.259313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:06.259524 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:06.260041 (+ 517us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:06.260202 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.261357 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.262840 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.263597 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.264038 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:06.264237 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:06.264289 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 01:06:06.264411 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":145,"threads_started":1}
I20260929 01:06:06.267727 18217 tablet_service.cc:1467] Processing CreateTablet for tablet a2c73f227dbf4cfba80889a25b707582 (DEFAULT_TABLE table=test-table [id=452643d6664b4b98814e761a36278f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:06:06.267650 18348 tablet_service.cc:1467] Processing CreateTablet for tablet a2c73f227dbf4cfba80889a25b707582 (DEFAULT_TABLE table=test-table [id=452643d6664b4b98814e761a36278f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:06:06.267648 18086 tablet_service.cc:1467] Processing CreateTablet for tablet a2c73f227dbf4cfba80889a25b707582 (DEFAULT_TABLE table=test-table [id=452643d6664b4b98814e761a36278f0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:06:06.268653 18217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2c73f227dbf4cfba80889a25b707582. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:06.268678 18086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2c73f227dbf4cfba80889a25b707582. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:06.268685 18348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2c73f227dbf4cfba80889a25b707582. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:06.275408 18431 tablet_bootstrap.cc:492] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a: Bootstrap starting.
I20260929 01:06:06.276379 18433 tablet_bootstrap.cc:492] T a2c73f227dbf4cfba80889a25b707582 P b80dba40734745db85ab077810cabd96: Bootstrap starting.
I20260929 01:06:06.277619 18431 tablet_bootstrap.cc:654] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:06.278030 18433 tablet_bootstrap.cc:654] T a2c73f227dbf4cfba80889a25b707582 P b80dba40734745db85ab077810cabd96: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:06.278411 18431 log.cc:824] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:06.278731 18433 log.cc:824] T a2c73f227dbf4cfba80889a25b707582 P b80dba40734745db85ab077810cabd96: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:06.279218 18432 tablet_bootstrap.cc:492] T a2c73f227dbf4cfba80889a25b707582 P f34cd0eb2c7d493fb3b3d0b3782ad1a6: Bootstrap starting.
I20260929 01:06:06.280483 18431 tablet_bootstrap.cc:492] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a: No bootstrap required, opened a new log
I20260929 01:06:06.280684 18431 ts_tablet_manager.cc:1397] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 01:06:06.281234 18432 tablet_bootstrap.cc:654] T a2c73f227dbf4cfba80889a25b707582 P f34cd0eb2c7d493fb3b3d0b3782ad1a6: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:06.281826 18432 log.cc:824] T a2c73f227dbf4cfba80889a25b707582 P f34cd0eb2c7d493fb3b3d0b3782ad1a6: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:06.283583 18432 tablet_bootstrap.cc:492] T a2c73f227dbf4cfba80889a25b707582 P f34cd0eb2c7d493fb3b3d0b3782ad1a6: No bootstrap required, opened a new log
I20260929 01:06:06.283797 18432 ts_tablet_manager.cc:1397] T a2c73f227dbf4cfba80889a25b707582 P f34cd0eb2c7d493fb3b3d0b3782ad1a6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 01:06:06.284044 18433 tablet_bootstrap.cc:492] T a2c73f227dbf4cfba80889a25b707582 P b80dba40734745db85ab077810cabd96: No bootstrap required, opened a new log
I20260929 01:06:06.284266 18433 ts_tablet_manager.cc:1397] T a2c73f227dbf4cfba80889a25b707582 P b80dba40734745db85ab077810cabd96: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260929 01:06:06.284713 18431 raft_consensus.cc:348] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28b86860c1c43d29b36e297cd280f8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "b80dba40734745db85ab077810cabd96" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44877 } } peers { permanent_uuid: "f34cd0eb2c7d493fb3b3d0b3782ad1a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42671 } }
I20260929 01:06:06.284988 18431 raft_consensus.cc:374] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:06.285048 18431 raft_consensus.cc:729] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b28b86860c1c43d29b36e297cd280f8a, State: Initialized, Role: FOLLOWER
I20260929 01:06:06.285487 18431 consensus_queue.cc:260] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28b86860c1c43d29b36e297cd280f8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "b80dba40734745db85ab077810cabd96" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44877 } } peers { permanent_uuid: "f34cd0eb2c7d493fb3b3d0b3782ad1a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42671 } }
I20260929 01:06:06.286211 18283 heartbeater.cc:503] Master 127.0.98.62:37983 was elected leader, sending a full tablet report...
I20260929 01:06:06.286582 18431 ts_tablet_manager.cc:1428] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 01:06:06.288194 18432 raft_consensus.cc:348] T a2c73f227dbf4cfba80889a25b707582 P f34cd0eb2c7d493fb3b3d0b3782ad1a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28b86860c1c43d29b36e297cd280f8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "b80dba40734745db85ab077810cabd96" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44877 } } peers { permanent_uuid: "f34cd0eb2c7d493fb3b3d0b3782ad1a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42671 } }
I20260929 01:06:06.288460 18432 raft_consensus.cc:374] T a2c73f227dbf4cfba80889a25b707582 P f34cd0eb2c7d493fb3b3d0b3782ad1a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:06.288528 18432 raft_consensus.cc:729] T a2c73f227dbf4cfba80889a25b707582 P f34cd0eb2c7d493fb3b3d0b3782ad1a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f34cd0eb2c7d493fb3b3d0b3782ad1a6, State: Initialized, Role: FOLLOWER
I20260929 01:06:06.288416 18433 raft_consensus.cc:348] T a2c73f227dbf4cfba80889a25b707582 P b80dba40734745db85ab077810cabd96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28b86860c1c43d29b36e297cd280f8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "b80dba40734745db85ab077810cabd96" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44877 } } peers { permanent_uuid: "f34cd0eb2c7d493fb3b3d0b3782ad1a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42671 } }
I20260929 01:06:06.288664 18433 raft_consensus.cc:374] T a2c73f227dbf4cfba80889a25b707582 P b80dba40734745db85ab077810cabd96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:06.288738 18433 raft_consensus.cc:729] T a2c73f227dbf4cfba80889a25b707582 P b80dba40734745db85ab077810cabd96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b80dba40734745db85ab077810cabd96, State: Initialized, Role: FOLLOWER
I20260929 01:06:06.288992 18432 consensus_queue.cc:260] T a2c73f227dbf4cfba80889a25b707582 P f34cd0eb2c7d493fb3b3d0b3782ad1a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28b86860c1c43d29b36e297cd280f8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "b80dba40734745db85ab077810cabd96" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44877 } } peers { permanent_uuid: "f34cd0eb2c7d493fb3b3d0b3782ad1a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42671 } }
I20260929 01:06:06.289152 18433 consensus_queue.cc:260] T a2c73f227dbf4cfba80889a25b707582 P b80dba40734745db85ab077810cabd96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28b86860c1c43d29b36e297cd280f8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "b80dba40734745db85ab077810cabd96" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44877 } } peers { permanent_uuid: "f34cd0eb2c7d493fb3b3d0b3782ad1a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42671 } }
I20260929 01:06:06.289924 18433 ts_tablet_manager.cc:1428] T a2c73f227dbf4cfba80889a25b707582 P b80dba40734745db85ab077810cabd96: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 01:06:06.290006 18432 ts_tablet_manager.cc:1428] T a2c73f227dbf4cfba80889a25b707582 P f34cd0eb2c7d493fb3b3d0b3782ad1a6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 01:06:06.290259 18152 heartbeater.cc:503] Master 127.0.98.62:37983 was elected leader, sending a full tablet report...
I20260929 01:06:06.290326 18414 heartbeater.cc:503] Master 127.0.98.62:37983 was elected leader, sending a full tablet report...
W20260929 01:06:06.322088 18153 tablet.cc:2369] T a2c73f227dbf4cfba80889a25b707582 P b80dba40734745db85ab077810cabd96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:06:06.356858 18437 raft_consensus.cc:482] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:06:06.357154 18437 raft_consensus.cc:504] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28b86860c1c43d29b36e297cd280f8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "b80dba40734745db85ab077810cabd96" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44877 } } peers { permanent_uuid: "f34cd0eb2c7d493fb3b3d0b3782ad1a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42671 } }
I20260929 01:06:06.358278 18437 leader_election.cc:288] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b80dba40734745db85ab077810cabd96 (127.0.98.1:44877), f34cd0eb2c7d493fb3b3d0b3782ad1a6 (127.0.98.3:42671)
I20260929 01:06:06.362159 18285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:06.358504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42671 (local address 127.0.98.2:49155)
0929 01:06:06.358729 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:06.358745 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:06.358869 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:06.359182 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:06.359184 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:06.359237 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:06.359362 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:06.359371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.360186 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.360189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:06.361112 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:06.361121 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.361740 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.361746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:06.361845 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:06.361875 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 01:06:06.361948 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 01:06:06.362001 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":55}
I20260929 01:06:06.362190 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:06.358598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49155 (local address 127.0.98.3:42671)
0929 01:06:06.358736 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:06.358741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:06.358815 (+ 74us) server_negotiation.cc:407] Connection header received
0929 01:06:06.359005 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:06.359010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:06.359075 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:06.359157 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:06.359482 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.360078 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.361241 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.361622 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.361713 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:06.361802 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:06.361986 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 01:06:06.362067 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 01:06:06.362269 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:06.358598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34885 (local address 127.0.98.1:44877)
0929 01:06:06.358728 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:06.358734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:06.359103 (+ 369us) server_negotiation.cc:407] Connection header received
0929 01:06:06.359260 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:06.359263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:06.359316 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:06.359403 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:06.359700 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.360229 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.361273 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.361653 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.361790 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:06.361851 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:06.362075 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 01:06:06.362143 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 01:06:06.362491 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:06.358504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44877 (local address 127.0.98.2:34885)
0929 01:06:06.358995 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:06.359008 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:06.359131 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:06.359438 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:06.359443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:06.359464 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:06.359593 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:06.359599 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.360338 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.360340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:06.361120 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:06.361128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.361823 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.361827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:06.361921 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:06.361967 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 01:06:06.362043 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 01:06:06.362103 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":79,"thread_start_us":129,"threads_started":1}
I20260929 01:06:06.362962 18368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c73f227dbf4cfba80889a25b707582" candidate_uuid: "b28b86860c1c43d29b36e297cd280f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f34cd0eb2c7d493fb3b3d0b3782ad1a6" is_pre_election: true
I20260929 01:06:06.363090 18106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c73f227dbf4cfba80889a25b707582" candidate_uuid: "b28b86860c1c43d29b36e297cd280f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b80dba40734745db85ab077810cabd96" is_pre_election: true
I20260929 01:06:06.363308 18368 raft_consensus.cc:2454] T a2c73f227dbf4cfba80889a25b707582 P f34cd0eb2c7d493fb3b3d0b3782ad1a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b28b86860c1c43d29b36e297cd280f8a in term 0.
I20260929 01:06:06.363353 18106 raft_consensus.cc:2454] T a2c73f227dbf4cfba80889a25b707582 P b80dba40734745db85ab077810cabd96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b28b86860c1c43d29b36e297cd280f8a in term 0.
I20260929 01:06:06.363830 18172 leader_election.cc:302] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b28b86860c1c43d29b36e297cd280f8a, b80dba40734745db85ab077810cabd96, f34cd0eb2c7d493fb3b3d0b3782ad1a6; no voters:
I20260929 01:06:06.364125 18437 raft_consensus.cc:2781] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:06:06.364224 18437 raft_consensus.cc:482] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:06:06.364282 18437 raft_consensus.cc:3037] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:06.365288 18437 raft_consensus.cc:504] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28b86860c1c43d29b36e297cd280f8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "b80dba40734745db85ab077810cabd96" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44877 } } peers { permanent_uuid: "f34cd0eb2c7d493fb3b3d0b3782ad1a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42671 } }
I20260929 01:06:06.365628 18437 leader_election.cc:288] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a [CANDIDATE]: Term 1 election: Requested vote from peers b80dba40734745db85ab077810cabd96 (127.0.98.1:44877), f34cd0eb2c7d493fb3b3d0b3782ad1a6 (127.0.98.3:42671)
I20260929 01:06:06.365950 18106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c73f227dbf4cfba80889a25b707582" candidate_uuid: "b28b86860c1c43d29b36e297cd280f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b80dba40734745db85ab077810cabd96"
I20260929 01:06:06.365957 18368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c73f227dbf4cfba80889a25b707582" candidate_uuid: "b28b86860c1c43d29b36e297cd280f8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f34cd0eb2c7d493fb3b3d0b3782ad1a6"
I20260929 01:06:06.366055 18106 raft_consensus.cc:3037] T a2c73f227dbf4cfba80889a25b707582 P b80dba40734745db85ab077810cabd96 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:06.366091 18368 raft_consensus.cc:3037] T a2c73f227dbf4cfba80889a25b707582 P f34cd0eb2c7d493fb3b3d0b3782ad1a6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:06.367178 18106 raft_consensus.cc:2454] T a2c73f227dbf4cfba80889a25b707582 P b80dba40734745db85ab077810cabd96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b28b86860c1c43d29b36e297cd280f8a in term 1.
I20260929 01:06:06.367288 18368 raft_consensus.cc:2454] T a2c73f227dbf4cfba80889a25b707582 P f34cd0eb2c7d493fb3b3d0b3782ad1a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b28b86860c1c43d29b36e297cd280f8a in term 1.
I20260929 01:06:06.367522 18172 leader_election.cc:302] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b28b86860c1c43d29b36e297cd280f8a, b80dba40734745db85ab077810cabd96; no voters:
I20260929 01:06:06.367683 18437 raft_consensus.cc:2781] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:06.367897 18437 raft_consensus.cc:686] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a [term 1 LEADER]: Becoming Leader. State: Replica: b28b86860c1c43d29b36e297cd280f8a, State: Running, Role: LEADER
I20260929 01:06:06.368227 18437 consensus_queue.cc:237] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28b86860c1c43d29b36e297cd280f8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } } peers { permanent_uuid: "b80dba40734745db85ab077810cabd96" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44877 } } peers { permanent_uuid: "f34cd0eb2c7d493fb3b3d0b3782ad1a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42671 } }
I20260929 01:06:06.371304 17964 catalog_manager.cc:5495] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a reported cstate change: term changed from 0 to 1, leader changed from <none> to b28b86860c1c43d29b36e297cd280f8a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b28b86860c1c43d29b36e297cd280f8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b28b86860c1c43d29b36e297cd280f8a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b80dba40734745db85ab077810cabd96" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f34cd0eb2c7d493fb3b3d0b3782ad1a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42671 } health_report { overall_health: UNKNOWN } } }
I20260929 01:06:06.385434 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:06.382138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60986 (local address 127.0.98.2:32937)
0929 01:06:06.382264 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:06.382267 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:06.382340 (+ 73us) server_negotiation.cc:407] Connection header received
0929 01:06:06.382465 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:06.382467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:06.382518 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:06.382600 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:06.382834 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.383336 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.384137 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:06.384352 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:06.384426 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:06.384721 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:06.385107 (+ 386us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:06.385199 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:06.385276 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 01:06:06.385319 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 01:06:06.391345 18106 raft_consensus.cc:1261] T a2c73f227dbf4cfba80889a25b707582 P b80dba40734745db85ab077810cabd96 [term 1 FOLLOWER]: Refusing update from remote peer b28b86860c1c43d29b36e297cd280f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:06:06.391345 18368 raft_consensus.cc:1261] T a2c73f227dbf4cfba80889a25b707582 P f34cd0eb2c7d493fb3b3d0b3782ad1a6 [term 1 FOLLOWER]: Refusing update from remote peer b28b86860c1c43d29b36e297cd280f8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:06:06.392026 18437 consensus_queue.cc:1035] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b80dba40734745db85ab077810cabd96" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:06:06.392277 18441 consensus_queue.cc:1035] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f34cd0eb2c7d493fb3b3d0b3782ad1a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:06:06.402467 18446 mvcc.cc:205] Tried to move back new op lower bound from 7334477686332108800 to 7334477686247006208. Current Snapshot: MvccSnapshot[applied={T|T < 7334477686332108800 or (T in {7334477686332108800})}]
I20260929 01:06:06.406562 17961 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:06:06.406739 17961 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:06:06.409014 17961 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d36e24df26247d6b9b5dc77545ad380: Sending DeleteTablet for 3 replicas of tablet a2c73f227dbf4cfba80889a25b707582
I20260929 01:06:06.409772 18086 tablet_service.cc:1514] Processing DeleteTablet for tablet a2c73f227dbf4cfba80889a25b707582 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:06:06 UTC) from {username='slave'} at 127.0.0.1:45512
I20260929 01:06:06.409816 18348 tablet_service.cc:1514] Processing DeleteTablet for tablet a2c73f227dbf4cfba80889a25b707582 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:06:06 UTC) from {username='slave'} at 127.0.0.1:57720
I20260929 01:06:06.410037 18217 tablet_service.cc:1514] Processing DeleteTablet for tablet a2c73f227dbf4cfba80889a25b707582 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:06:06 UTC) from {username='slave'} at 127.0.0.1:60984
I20260929 01:06:06.410424 18455 tablet_replica.cc:330] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a: stopping tablet replica
I20260929 01:06:06.410657 18453 tablet_replica.cc:330] T a2c73f227dbf4cfba80889a25b707582 P b80dba40734745db85ab077810cabd96: stopping tablet replica
I20260929 01:06:06.410737 18455 raft_consensus.cc:2229] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:06:06.410948 18453 raft_consensus.cc:2229] T a2c73f227dbf4cfba80889a25b707582 P b80dba40734745db85ab077810cabd96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:06:06.411077 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 18024
I20260929 01:06:06.411099 18455 raft_consensus.cc:2258] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:06:06.411201 18453 raft_consensus.cc:2258] T a2c73f227dbf4cfba80889a25b707582 P b80dba40734745db85ab077810cabd96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:06:06.411662 18454 tablet_replica.cc:330] T a2c73f227dbf4cfba80889a25b707582 P f34cd0eb2c7d493fb3b3d0b3782ad1a6: stopping tablet replica
I20260929 01:06:06.412212 18454 raft_consensus.cc:2229] T a2c73f227dbf4cfba80889a25b707582 P f34cd0eb2c7d493fb3b3d0b3782ad1a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:06:06.412479 18454 raft_consensus.cc:2258] T a2c73f227dbf4cfba80889a25b707582 P f34cd0eb2c7d493fb3b3d0b3782ad1a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:06:06.413877 18454 ts_tablet_manager.cc:1905] T a2c73f227dbf4cfba80889a25b707582 P f34cd0eb2c7d493fb3b3d0b3782ad1a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:06:06.416018 18454 ts_tablet_manager.cc:1918] T a2c73f227dbf4cfba80889a25b707582 P f34cd0eb2c7d493fb3b3d0b3782ad1a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:06:06.416124 18454 log.cc:1196] T a2c73f227dbf4cfba80889a25b707582 P f34cd0eb2c7d493fb3b3d0b3782ad1a6: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/a2c73f227dbf4cfba80889a25b707582
I20260929 01:06:06.416425 18454 ts_tablet_manager.cc:1939] T a2c73f227dbf4cfba80889a25b707582 P f34cd0eb2c7d493fb3b3d0b3782ad1a6: Deleting consensus metadata
I20260929 01:06:06.417613 18455 ts_tablet_manager.cc:1905] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:06:06.417608 17951 catalog_manager.cc:4841] TS f34cd0eb2c7d493fb3b3d0b3782ad1a6 (127.0.98.3:42671): tablet a2c73f227dbf4cfba80889a25b707582 (table test-table [id=452643d6664b4b98814e761a36278f0c]) successfully deleted
W20260929 01:06:06.419286 17950 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44877 (error 108)
I20260929 01:06:06.419998 18455 ts_tablet_manager.cc:1918] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:06:06.420104 18455 log.cc:1196] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790643720886468-392-0/minicluster-data/ts-1/wal/wals/a2c73f227dbf4cfba80889a25b707582
I20260929 01:06:06.420369 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:06.420015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44877 (local address 127.0.0.1:45522)
0929 01:06:06.420179 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:06.420283 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 01:06:06.420382 18455 ts_tablet_manager.cc:1939] T a2c73f227dbf4cfba80889a25b707582 P b28b86860c1c43d29b36e297cd280f8a: Deleting consensus metadata
W20260929 01:06:06.420536 17950 catalog_manager.cc:4584] TS b80dba40734745db85ab077810cabd96 (127.0.98.1:44877): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a2c73f227dbf4cfba80889a25b707582: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44877: connect: Connection refused (error 111)
I20260929 01:06:06.420928 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 18155
I20260929 01:06:06.426529 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 18286
I20260929 01:06:06.426620 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:06.426348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32937 (local address 127.0.0.1:60990)
0929 01:06:06.426468 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:06.426522 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:32937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 01:06:06.431975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 17933
2026-09-29T01:06:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1068 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:06 dist-test-slave-7zdl krb5kdc[18459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:06 dist-test-slave-7zdl krb5kdc[18459](info): set up 2 sockets
Sep 29 01:06:06 dist-test-slave-7zdl krb5kdc[18459](info): commencing operation
krb5kdc: starting...
W20260929 01:06:07.466759 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:06:07 dist-test-slave-7zdl krb5kdc[18459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:06:07.585958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33389
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:07.693369 18472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:07.693655 18472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:07.693734 18472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:07.697652 18472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:07.697731 18472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:07.697768 18472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:07.697826 18472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:07.697865 18472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:07.702976 18472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:07.703943 18472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:07.705111 18472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:07.715879 18478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:07.715865 18480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:07.715889 18477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:07.716166 18472 server_base.cc:1034] running on GCE node
I20260929 01:06:07.717067 18472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:07.718128 18472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:07.719321 18472 hybrid_clock.cc:648] HybridClock initialized: now 1790643967719306 us; error 28 us; skew 500 ppm
Sep 29 01:06:07 dist-test-slave-7zdl krb5kdc[18459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:07.721973 18472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:06:07.722880 18472 webserver.cc:460] Webserver started at http://127.0.98.62:41469/ using document root <none> and password file <none>
I20260929 01:06:07.723433 18472 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:07.723529 18472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:07.723759 18472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:07.725723 18472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "24cb8b84367e403ca375b98c5f3176af"
format_stamp: "Formatted at 2026-09-29 01:06:07 on dist-test-slave-7zdl"
server_key: "a3fd2e0f7e481e8644862bfc415af15a"
server_key_iv: "0c16e86610766721c977e7bb4df332a5"
server_key_version: "encryptionkey@0"
I20260929 01:06:07.726292 18472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "24cb8b84367e403ca375b98c5f3176af"
format_stamp: "Formatted at 2026-09-29 01:06:07 on dist-test-slave-7zdl"
server_key: "a3fd2e0f7e481e8644862bfc415af15a"
server_key_iv: "0c16e86610766721c977e7bb4df332a5"
server_key_version: "encryptionkey@0"
I20260929 01:06:07.729887 18472 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 01:06:07.732331 18487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:07.733548 18472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:06:07.733695 18472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "24cb8b84367e403ca375b98c5f3176af"
format_stamp: "Formatted at 2026-09-29 01:06:07 on dist-test-slave-7zdl"
server_key: "a3fd2e0f7e481e8644862bfc415af15a"
server_key_iv: "0c16e86610766721c977e7bb4df332a5"
server_key_version: "encryptionkey@0"
I20260929 01:06:07.733814 18472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:07.745510 18472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:07.748324 18472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:07.748514 18472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:07.755810 18472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33389
I20260929 01:06:07.755834 18539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33389 every 8 connection(s)
I20260929 01:06:07.756697 18472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:07.759394 18540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:07.762032 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 18472
I20260929 01:06:07.762192 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:07.762542 392 external_mini_cluster.cc:1428] Setting key 89d70425546234ac6eac01d66b70db70
I20260929 01:06:07.765865 18540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24cb8b84367e403ca375b98c5f3176af: Bootstrap starting.
I20260929 01:06:07.768472 18540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24cb8b84367e403ca375b98c5f3176af: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:07.769266 18540 log.cc:824] T 00000000000000000000000000000000 P 24cb8b84367e403ca375b98c5f3176af: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:07.771206 18540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24cb8b84367e403ca375b98c5f3176af: No bootstrap required, opened a new log
Sep 29 01:06:07 dist-test-slave-7zdl krb5kdc[18459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:06:07.774277 18540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24cb8b84367e403ca375b98c5f3176af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24cb8b84367e403ca375b98c5f3176af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33389 } }
I20260929 01:06:07.774521 18540 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24cb8b84367e403ca375b98c5f3176af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:07.774614 18540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24cb8b84367e403ca375b98c5f3176af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24cb8b84367e403ca375b98c5f3176af, State: Initialized, Role: FOLLOWER
I20260929 01:06:07.775079 18540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24cb8b84367e403ca375b98c5f3176af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24cb8b84367e403ca375b98c5f3176af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33389 } }
I20260929 01:06:07.775231 18540 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24cb8b84367e403ca375b98c5f3176af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:07.775317 18540 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24cb8b84367e403ca375b98c5f3176af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:07.775413 18540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24cb8b84367e403ca375b98c5f3176af [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:07.776275 18540 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24cb8b84367e403ca375b98c5f3176af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24cb8b84367e403ca375b98c5f3176af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33389 } }
I20260929 01:06:07.776630 18540 leader_election.cc:302] T 00000000000000000000000000000000 P 24cb8b84367e403ca375b98c5f3176af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24cb8b84367e403ca375b98c5f3176af; no voters:
I20260929 01:06:07.776957 18540 leader_election.cc:288] T 00000000000000000000000000000000 P 24cb8b84367e403ca375b98c5f3176af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:07.777083 18545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24cb8b84367e403ca375b98c5f3176af [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:07.777310 18545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24cb8b84367e403ca375b98c5f3176af [term 1 LEADER]: Becoming Leader. State: Replica: 24cb8b84367e403ca375b98c5f3176af, State: Running, Role: LEADER
I20260929 01:06:07.777683 18545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24cb8b84367e403ca375b98c5f3176af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24cb8b84367e403ca375b98c5f3176af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33389 } }
I20260929 01:06:07.778064 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:07.764091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33326 (local address 127.0.98.62:33389)
0929 01:06:07.764585 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:07.764604 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:07.764646 (+ 42us) server_negotiation.cc:407] Connection header received
0929 01:06:07.765434 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:07.765454 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:07.765837 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:07.766184 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:07.766872 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:07.768360 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:07.769225 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:07.769819 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:07.770002 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:07.772466 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:07.772470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:07.772484 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:07.772508 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:07.774935 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:07.775393 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:07.775399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:07.775404 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:07.775559 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:07.775823 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:07.775827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:07.775828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:07.776245 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:07.776473 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:07.776756 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 01:06:07.777082 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":144,"threads_started":1}
I20260929 01:06:07.778239 18540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24cb8b84367e403ca375b98c5f3176af [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:07.779276 18546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24cb8b84367e403ca375b98c5f3176af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24cb8b84367e403ca375b98c5f3176af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24cb8b84367e403ca375b98c5f3176af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33389 } } }
I20260929 01:06:07.779387 18546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24cb8b84367e403ca375b98c5f3176af [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:07.779595 18546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24cb8b84367e403ca375b98c5f3176af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24cb8b84367e403ca375b98c5f3176af. Latest consensus state: current_term: 1 leader_uuid: "24cb8b84367e403ca375b98c5f3176af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24cb8b84367e403ca375b98c5f3176af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33389 } } }
I20260929 01:06:07.779664 18546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24cb8b84367e403ca375b98c5f3176af [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:07.780319 18554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:07.783257 18554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:07.788749 18554 catalog_manager.cc:1347] Generated new cluster ID: 2e6691cef0b941f5a401adb5f69859c9
I20260929 01:06:07.788828 18554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:07.803520 18554 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:07.804503 18554 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:07.809515 18554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24cb8b84367e403ca375b98c5f3176af: Generated new TSK 0
I20260929 01:06:07.810179 18554 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 01:06:07.842415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33389
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:07.952014 18566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:07.952291 18566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:07.952332 18566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:07.956396 18566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:07.956449 18566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:07.956549 18566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:06:07.961651 18566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:07.962659 18566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:07.963721 18566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:07.973304 18572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:07.973280 18574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:07.973292 18571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:07.973358 18566 server_base.cc:1034] running on GCE node
I20260929 01:06:07.973891 18566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:07.974507 18566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:07.975687 18566 hybrid_clock.cc:648] HybridClock initialized: now 1790643967975656 us; error 53 us; skew 500 ppm
Sep 29 01:06:07 dist-test-slave-7zdl krb5kdc[18459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:07.978503 18566 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:06:07.979362 18566 webserver.cc:460] Webserver started at http://127.0.98.1:35863/ using document root <none> and password file <none>
I20260929 01:06:07.979987 18566 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:07.980048 18566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:07.980218 18566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:07.982128 18566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "2a35a86ffa7b45b495ec63ce2e59bdaf"
format_stamp: "Formatted at 2026-09-29 01:06:07 on dist-test-slave-7zdl"
server_key: "911dcfb42077b666a62c4b2089bde8e5"
server_key_iv: "ffcb1ea8ab5c00fd6f912da2ea3762c3"
server_key_version: "encryptionkey@0"
I20260929 01:06:07.982617 18566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2a35a86ffa7b45b495ec63ce2e59bdaf"
format_stamp: "Formatted at 2026-09-29 01:06:07 on dist-test-slave-7zdl"
server_key: "911dcfb42077b666a62c4b2089bde8e5"
server_key_iv: "ffcb1ea8ab5c00fd6f912da2ea3762c3"
server_key_version: "encryptionkey@0"
I20260929 01:06:07.986158 18566 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:06:07.988485 18581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:07.989655 18566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:06:07.989770 18566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "2a35a86ffa7b45b495ec63ce2e59bdaf"
format_stamp: "Formatted at 2026-09-29 01:06:07 on dist-test-slave-7zdl"
server_key: "911dcfb42077b666a62c4b2089bde8e5"
server_key_iv: "ffcb1ea8ab5c00fd6f912da2ea3762c3"
server_key_version: "encryptionkey@0"
I20260929 01:06:07.989871 18566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:08.003829 18566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:08.006621 18566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:08.006872 18566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:08.007452 18566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:06:08.008298 18566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:06:08.008344 18566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:08.008410 18566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:06:08.008440 18566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:08.017169 18566 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42597
I20260929 01:06:08.017236 18694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42597 every 8 connection(s)
I20260929 01:06:08.018013 18566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:06:08.018484 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 18566
I20260929 01:06:08.018615 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:06:08.018962 392 external_mini_cluster.cc:1428] Setting key bb37e59e0a5d9c4c8c06610aa397c2cf
Sep 29 01:06:08 dist-test-slave-7zdl krb5kdc[18459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 01:06:08.033219 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:08.019825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45947 (local address 127.0.98.62:33389)
0929 01:06:08.019950 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:08.019954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:08.021049 (+ 1095us) server_negotiation.cc:407] Connection header received
0929 01:06:08.021990 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:08.021997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:08.022067 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:08.022157 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:08.023657 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.024529 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.026361 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.026592 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.026679 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:08.029558 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:08.029566 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:08.029568 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:08.029592 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:08.031064 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:08.031571 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:08.031575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:08.031577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:08.031657 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:08.031988 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:08.031992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:08.031993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:08.032255 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:08.032357 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:08.032912 (+ 555us) server_negotiation.cc:299] Negotiation successful
0929 01:06:08.033075 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 01:06:08.034238 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:08.020363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33389 (local address 127.0.98.1:45947)
0929 01:06:08.020886 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:08.020937 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:08.021789 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:08.022313 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:08.022319 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:08.023016 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:08.023500 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:08.023519 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.024664 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.024670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:08.026235 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:08.026251 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.026758 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.026765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:08.027024 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:08.027688 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:08.027712 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:08.029397 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:08.031203 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:08.031209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:08.031222 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:08.031461 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:08.031757 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:08.031760 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:08.031762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:08.031891 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:08.032366 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:08.032375 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:08.032655 (+ 280us) client_negotiation.cc:769] Sending connection context
0929 01:06:08.032841 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 01:06:08.033137 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":146,"threads_started":1}
I20260929 01:06:08.035548 18695 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33389
I20260929 01:06:08.035802 18695 heartbeater.cc:464] Registering TS with master...
I20260929 01:06:08.036368 18695 heartbeater.cc:511] Master 127.0.98.62:33389 requested a full tablet report, sending...
I20260929 01:06:08.038092 18504 ts_manager.cc:194] Registered new tserver with Master: 2a35a86ffa7b45b495ec63ce2e59bdaf (127.0.98.1:42597)
I20260929 01:06:08.039296 18504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 01:06:08.047643 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33389
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:08.154522 18700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:08.154801 18700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:08.154899 18700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:08.158814 18700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:08.158900 18700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:08.158988 18700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:06:08.164031 18700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:08.165028 18700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:08.166009 18700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:08.175554 18708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:08.175602 18700 server_base.cc:1034] running on GCE node
W20260929 01:06:08.175614 18706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:08.175606 18705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:08.176308 18700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:08.176923 18700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:08.178107 18700 hybrid_clock.cc:648] HybridClock initialized: now 1790643968178084 us; error 26 us; skew 500 ppm
Sep 29 01:06:08 dist-test-slave-7zdl krb5kdc[18459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:08.186944 18700 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:06:08.187847 18700 webserver.cc:460] Webserver started at http://127.0.98.2:37337/ using document root <none> and password file <none>
I20260929 01:06:08.188398 18700 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:08.188513 18700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:08.188732 18700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:08.190748 18700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "048ffc2857be456f85d61b8d75cf6bfa"
format_stamp: "Formatted at 2026-09-29 01:06:08 on dist-test-slave-7zdl"
server_key: "f7e2f47368a6ca3184b34da0a886076c"
server_key_iv: "701fc38861146efb260c22b5f12343d5"
server_key_version: "encryptionkey@0"
I20260929 01:06:08.191347 18700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "048ffc2857be456f85d61b8d75cf6bfa"
format_stamp: "Formatted at 2026-09-29 01:06:08 on dist-test-slave-7zdl"
server_key: "f7e2f47368a6ca3184b34da0a886076c"
server_key_iv: "701fc38861146efb260c22b5f12343d5"
server_key_version: "encryptionkey@0"
I20260929 01:06:08.195004 18700 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:06:08.197232 18715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:08.198428 18700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:06:08.198580 18700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "048ffc2857be456f85d61b8d75cf6bfa"
format_stamp: "Formatted at 2026-09-29 01:06:08 on dist-test-slave-7zdl"
server_key: "f7e2f47368a6ca3184b34da0a886076c"
server_key_iv: "701fc38861146efb260c22b5f12343d5"
server_key_version: "encryptionkey@0"
I20260929 01:06:08.198698 18700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:08.217406 18700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:08.220603 18700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:08.220809 18700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:08.221375 18700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:06:08.222165 18700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:06:08.222244 18700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:08.222311 18700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:06:08.222359 18700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:08.231277 18700 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46003
I20260929 01:06:08.231321 18828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46003 every 8 connection(s)
I20260929 01:06:08.232223 18700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:06:08.233278 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 18700
I20260929 01:06:08.233405 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:06:08.233767 392 external_mini_cluster.cc:1428] Setting key ddc8de59428ce01bae99678a82ac2d46
Sep 29 01:06:08 dist-test-slave-7zdl krb5kdc[18459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 01:06:08.245782 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:08.234169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46289 (local address 127.0.98.62:33389)
0929 01:06:08.234280 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:08.234284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:08.235322 (+ 1038us) server_negotiation.cc:407] Connection header received
0929 01:06:08.236208 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:08.236211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:08.236284 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:08.236367 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:08.237650 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.238248 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.239528 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.239771 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.239881 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:08.242237 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:08.242241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:08.242243 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:08.242275 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:08.243786 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:08.244286 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:08.244291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:08.244293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:08.244375 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:08.244733 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:08.244736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:08.244738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:08.244926 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:08.245032 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:08.245495 (+ 463us) server_negotiation.cc:299] Negotiation successful
0929 01:06:08.245611 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 01:06:08.246527 18831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:08.234625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33389 (local address 127.0.98.2:46289)
0929 01:06:08.235144 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:08.235214 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:08.235999 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:08.236525 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:08.236532 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:08.236886 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:08.237520 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:08.237536 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.238370 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.238376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:08.239404 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:08.239415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.239913 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.239917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:08.240097 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:08.240683 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:08.240698 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:08.242076 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:08.243917 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:08.243923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:08.243935 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:08.244184 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:08.244468 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:08.244473 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:08.244475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:08.244622 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:08.245029 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:08.245035 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:08.245266 (+ 231us) client_negotiation.cc:769] Sending connection context
0929 01:06:08.245441 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 01:06:08.245750 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":159,"threads_started":1}
I20260929 01:06:08.247750 18829 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33389
I20260929 01:06:08.248000 18829 heartbeater.cc:464] Registering TS with master...
I20260929 01:06:08.248560 18829 heartbeater.cc:511] Master 127.0.98.62:33389 requested a full tablet report, sending...
I20260929 01:06:08.249716 18504 ts_manager.cc:194] Registered new tserver with Master: 048ffc2857be456f85d61b8d75cf6bfa (127.0.98.2:46003)
I20260929 01:06:08.250618 18504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 01:06:08.261343 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33389
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:08.369521 18834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:08.369794 18834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:08.369882 18834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:08.374133 18834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:08.374217 18834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:08.374300 18834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:06:08.379680 18834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:08.380698 18834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:08.381688 18834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:08.391338 18839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:08.391374 18842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:08.391398 18834 server_base.cc:1034] running on GCE node
W20260929 01:06:08.391347 18840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:08.391961 18834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:08.392580 18834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:08.393743 18834 hybrid_clock.cc:648] HybridClock initialized: now 1790643968393705 us; error 45 us; skew 500 ppm
Sep 29 01:06:08 dist-test-slave-7zdl krb5kdc[18459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:08.396648 18834 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:06:08.397516 18834 webserver.cc:460] Webserver started at http://127.0.98.3:39705/ using document root <none> and password file <none>
I20260929 01:06:08.398087 18834 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:08.398176 18834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:08.398403 18834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:08.400462 18834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "77cea489a5e243059d501c5c3a7c0689"
format_stamp: "Formatted at 2026-09-29 01:06:08 on dist-test-slave-7zdl"
server_key: "94da7630a39f1987fe1e56376d930c52"
server_key_iv: "9d5160800cd73362ea2a63f0ef8f309e"
server_key_version: "encryptionkey@0"
I20260929 01:06:08.401031 18834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "77cea489a5e243059d501c5c3a7c0689"
format_stamp: "Formatted at 2026-09-29 01:06:08 on dist-test-slave-7zdl"
server_key: "94da7630a39f1987fe1e56376d930c52"
server_key_iv: "9d5160800cd73362ea2a63f0ef8f309e"
server_key_version: "encryptionkey@0"
I20260929 01:06:08.404819 18834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 01:06:08.407183 18849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:08.408298 18834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:06:08.408439 18834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "77cea489a5e243059d501c5c3a7c0689"
format_stamp: "Formatted at 2026-09-29 01:06:08 on dist-test-slave-7zdl"
server_key: "94da7630a39f1987fe1e56376d930c52"
server_key_iv: "9d5160800cd73362ea2a63f0ef8f309e"
server_key_version: "encryptionkey@0"
I20260929 01:06:08.408561 18834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:08.426468 18834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:08.429330 18834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:08.429538 18834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:08.430128 18834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:06:08.430964 18834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:06:08.431037 18834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:08.431105 18834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:06:08.431154 18834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:08.440227 18834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45105
I20260929 01:06:08.440250 18962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45105 every 8 connection(s)
I20260929 01:06:08.441080 18834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:06:08.447000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 18834
I20260929 01:06:08.447134 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:06:08.447427 392 external_mini_cluster.cc:1428] Setting key bef05c1a89b533add4347c1d47b92678
Sep 29 01:06:08 dist-test-slave-7zdl krb5kdc[18459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:06:08.454598 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:08.442826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56183 (local address 127.0.98.62:33389)
0929 01:06:08.442969 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:08.442973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:08.443808 (+ 835us) server_negotiation.cc:407] Connection header received
0929 01:06:08.444641 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:08.444644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:08.444692 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:08.444766 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:08.445869 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.446397 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.447674 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.447923 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.448009 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:08.450556 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:08.450559 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:08.450561 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:08.450584 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:08.452144 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:08.452698 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:08.452702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:08.452704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:08.452791 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:08.453195 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:08.453200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:08.453203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:08.453422 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:08.453519 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:08.454204 (+ 685us) server_negotiation.cc:299] Negotiation successful
0929 01:06:08.454374 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 01:06:08.455462 18965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:08.443193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33389 (local address 127.0.98.3:56183)
0929 01:06:08.443658 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:08.443698 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:08.444436 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:08.444914 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:08.444920 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:08.445253 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:08.445732 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:08.445745 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.446507 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.446510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:08.447528 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:08.447541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.448071 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.448077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:08.448273 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:08.448847 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:08.448863 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:08.450402 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:08.452280 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:08.452286 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:08.452300 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:08.452576 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:08.452889 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:08.452895 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:08.452897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:08.453087 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:08.453531 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:08.453539 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:08.453864 (+ 325us) client_negotiation.cc:769] Sending connection context
0929 01:06:08.454103 (+ 239us) client_negotiation.cc:240] Negotiation successful
0929 01:06:08.454418 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":108,"threads_started":1}
I20260929 01:06:08.456514 18963 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33389
I20260929 01:06:08.456769 18963 heartbeater.cc:464] Registering TS with master...
I20260929 01:06:08.457244 18963 heartbeater.cc:511] Master 127.0.98.62:33389 requested a full tablet report, sending...
I20260929 01:06:08.458280 18504 ts_manager.cc:194] Registered new tserver with Master: 77cea489a5e243059d501c5c3a7c0689 (127.0.98.3:45105)
I20260929 01:06:08.458935 18504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56183
I20260929 01:06:08.461169 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:06:08.470956 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:08.463228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33336 (local address 127.0.98.62:33389)
0929 01:06:08.463398 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:08.463403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:08.463567 (+ 164us) server_negotiation.cc:407] Connection header received
0929 01:06:08.463714 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:08.463718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:08.463761 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:08.463853 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:08.464582 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.465090 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.465921 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.466160 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.466266 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:08.467854 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:08.467858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:08.467861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:08.467884 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:08.469149 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:08.469658 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:08.469662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:08.469663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:08.469746 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:08.470071 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:08.470074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:08.470076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:08.470274 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:08.470433 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:08.470688 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 01:06:08.470774 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 01:06:08.473913 18504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:06:08.476919 18504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:06:08.492355 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:08.486384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42597 (local address 127.0.0.1:33540)
0929 01:06:08.487387 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:08.487402 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:08.487500 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:08.488203 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:08.488207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:08.488226 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:08.488307 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:08.488313 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.489489 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.489494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:08.490513 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:08.490524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.491789 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.491795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:08.491953 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:08.491999 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 01:06:08.492103 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 01:06:08.492211 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":935,"mutex_wait_us":25,"thread_start_us":103,"threads_started":1}
I20260929 01:06:08.492367 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:08.487013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33540 (local address 127.0.98.1:42597)
0929 01:06:08.487750 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:08.487766 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:08.487789 (+ 23us) server_negotiation.cc:407] Connection header received
0929 01:06:08.487860 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:08.487865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:08.488053 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:08.488212 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:08.488426 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.489356 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.490672 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.491567 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.491693 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:08.491842 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:08.492162 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 01:06:08.492243 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":105,"threads_started":1}
I20260929 01:06:08.493031 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:08.486101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45105 (local address 127.0.0.1:52748)
0929 01:06:08.486358 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:08.486370 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:08.486433 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:08.487528 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:08.487531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:08.487547 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:08.487640 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:08.487646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.489026 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.489029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:08.490520 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:08.490526 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.491725 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.492610 (+ 885us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:08.492730 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:08.492753 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 01:06:08.492822 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 01:06:08.492890 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":203,"thread_start_us":96,"threads_started":1}
I20260929 01:06:08.493063 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:08.485621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46003 (local address 127.0.0.1:36542)
0929 01:06:08.486215 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:08.486283 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:08.486411 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:08.487335 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:08.487341 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:08.487369 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:08.487536 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:08.487543 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.488892 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.488898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:08.490434 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:08.490442 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.492746 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.492750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:08.492840 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:08.492858 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 01:06:08.492898 (+ 40us) client_negotiation.cc:240] Negotiation successful
0929 01:06:08.492963 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":78,"thread_start_us":118,"threads_started":1}
I20260929 01:06:08.493475 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:08.486550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.0.98.3:45105)
0929 01:06:08.487057 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:08.487072 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:08.487098 (+ 26us) server_negotiation.cc:407] Connection header received
0929 01:06:08.487182 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:08.487188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:08.487368 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:08.487555 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:08.487725 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.488885 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.490672 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.491567 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.491693 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:08.491842 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:08.492880 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0929 01:06:08.492999 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":160,"threads_started":1}
I20260929 01:06:08.493711 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:08.485735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36542 (local address 127.0.98.2:46003)
0929 01:06:08.486786 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:08.486798 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:08.486840 (+ 42us) server_negotiation.cc:407] Connection header received
0929 01:06:08.486924 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:08.486932 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:08.487146 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:08.487690 (+ 544us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:08.487841 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.488771 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.490562 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.492624 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.493304 (+ 680us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:08.493428 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:08.493471 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 01:06:08.493547 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":927,"thread_start_us":120,"threads_started":1}
I20260929 01:06:08.495359 18629 tablet_service.cc:1467] Processing CreateTablet for tablet 79ae7bc9a0184e1cb8b183b51c63f629 (DEFAULT_TABLE table=test-table [id=06672ae6c3d6483fb1dcd41698c9db33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:06:08.496274 18629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79ae7bc9a0184e1cb8b183b51c63f629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:08.496209 18763 tablet_service.cc:1467] Processing CreateTablet for tablet 79ae7bc9a0184e1cb8b183b51c63f629 (DEFAULT_TABLE table=test-table [id=06672ae6c3d6483fb1dcd41698c9db33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:06:08.496210 18897 tablet_service.cc:1467] Processing CreateTablet for tablet 79ae7bc9a0184e1cb8b183b51c63f629 (DEFAULT_TABLE table=test-table [id=06672ae6c3d6483fb1dcd41698c9db33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:06:08.497201 18763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79ae7bc9a0184e1cb8b183b51c63f629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:08.497198 18897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79ae7bc9a0184e1cb8b183b51c63f629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:08.502486 18980 tablet_bootstrap.cc:492] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf: Bootstrap starting.
I20260929 01:06:08.504819 18982 tablet_bootstrap.cc:492] T 79ae7bc9a0184e1cb8b183b51c63f629 P 048ffc2857be456f85d61b8d75cf6bfa: Bootstrap starting.
I20260929 01:06:08.505136 18981 tablet_bootstrap.cc:492] T 79ae7bc9a0184e1cb8b183b51c63f629 P 77cea489a5e243059d501c5c3a7c0689: Bootstrap starting.
I20260929 01:06:08.505322 18980 tablet_bootstrap.cc:654] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:08.506330 18980 log.cc:824] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:08.507089 18981 tablet_bootstrap.cc:654] T 79ae7bc9a0184e1cb8b183b51c63f629 P 77cea489a5e243059d501c5c3a7c0689: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:08.507490 18982 tablet_bootstrap.cc:654] T 79ae7bc9a0184e1cb8b183b51c63f629 P 048ffc2857be456f85d61b8d75cf6bfa: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:08.508279 18980 tablet_bootstrap.cc:492] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf: No bootstrap required, opened a new log
I20260929 01:06:08.508324 18981 log.cc:824] T 79ae7bc9a0184e1cb8b183b51c63f629 P 77cea489a5e243059d501c5c3a7c0689: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:08.508324 18982 log.cc:824] T 79ae7bc9a0184e1cb8b183b51c63f629 P 048ffc2857be456f85d61b8d75cf6bfa: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:08.508467 18980 ts_tablet_manager.cc:1397] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 01:06:08.511407 18980 raft_consensus.cc:348] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048ffc2857be456f85d61b8d75cf6bfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "2a35a86ffa7b45b495ec63ce2e59bdaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42597 } } peers { permanent_uuid: "77cea489a5e243059d501c5c3a7c0689" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45105 } }
I20260929 01:06:08.511672 18980 raft_consensus.cc:374] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:08.511731 18980 raft_consensus.cc:729] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a35a86ffa7b45b495ec63ce2e59bdaf, State: Initialized, Role: FOLLOWER
I20260929 01:06:08.512272 18980 consensus_queue.cc:260] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048ffc2857be456f85d61b8d75cf6bfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "2a35a86ffa7b45b495ec63ce2e59bdaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42597 } } peers { permanent_uuid: "77cea489a5e243059d501c5c3a7c0689" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45105 } }
I20260929 01:06:08.512398 18982 tablet_bootstrap.cc:492] T 79ae7bc9a0184e1cb8b183b51c63f629 P 048ffc2857be456f85d61b8d75cf6bfa: No bootstrap required, opened a new log
I20260929 01:06:08.512614 18982 ts_tablet_manager.cc:1397] T 79ae7bc9a0184e1cb8b183b51c63f629 P 048ffc2857be456f85d61b8d75cf6bfa: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260929 01:06:08.512964 18695 heartbeater.cc:503] Master 127.0.98.62:33389 was elected leader, sending a full tablet report...
I20260929 01:06:08.513226 18980 ts_tablet_manager.cc:1428] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 01:06:08.516081 18981 tablet_bootstrap.cc:492] T 79ae7bc9a0184e1cb8b183b51c63f629 P 77cea489a5e243059d501c5c3a7c0689: No bootstrap required, opened a new log
I20260929 01:06:08.516120 18982 raft_consensus.cc:348] T 79ae7bc9a0184e1cb8b183b51c63f629 P 048ffc2857be456f85d61b8d75cf6bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048ffc2857be456f85d61b8d75cf6bfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "2a35a86ffa7b45b495ec63ce2e59bdaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42597 } } peers { permanent_uuid: "77cea489a5e243059d501c5c3a7c0689" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45105 } }
I20260929 01:06:08.516319 18981 ts_tablet_manager.cc:1397] T 79ae7bc9a0184e1cb8b183b51c63f629 P 77cea489a5e243059d501c5c3a7c0689: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260929 01:06:08.516374 18982 raft_consensus.cc:374] T 79ae7bc9a0184e1cb8b183b51c63f629 P 048ffc2857be456f85d61b8d75cf6bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:08.516431 18982 raft_consensus.cc:729] T 79ae7bc9a0184e1cb8b183b51c63f629 P 048ffc2857be456f85d61b8d75cf6bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048ffc2857be456f85d61b8d75cf6bfa, State: Initialized, Role: FOLLOWER
I20260929 01:06:08.516875 18982 consensus_queue.cc:260] T 79ae7bc9a0184e1cb8b183b51c63f629 P 048ffc2857be456f85d61b8d75cf6bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048ffc2857be456f85d61b8d75cf6bfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "2a35a86ffa7b45b495ec63ce2e59bdaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42597 } } peers { permanent_uuid: "77cea489a5e243059d501c5c3a7c0689" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45105 } }
I20260929 01:06:08.517474 18829 heartbeater.cc:503] Master 127.0.98.62:33389 was elected leader, sending a full tablet report...
I20260929 01:06:08.517805 18982 ts_tablet_manager.cc:1428] T 79ae7bc9a0184e1cb8b183b51c63f629 P 048ffc2857be456f85d61b8d75cf6bfa: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260929 01:06:08.519107 18696 tablet.cc:2369] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:06:08.519845 18981 raft_consensus.cc:348] T 79ae7bc9a0184e1cb8b183b51c63f629 P 77cea489a5e243059d501c5c3a7c0689 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048ffc2857be456f85d61b8d75cf6bfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "2a35a86ffa7b45b495ec63ce2e59bdaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42597 } } peers { permanent_uuid: "77cea489a5e243059d501c5c3a7c0689" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45105 } }
I20260929 01:06:08.520145 18981 raft_consensus.cc:374] T 79ae7bc9a0184e1cb8b183b51c63f629 P 77cea489a5e243059d501c5c3a7c0689 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:08.520210 18981 raft_consensus.cc:729] T 79ae7bc9a0184e1cb8b183b51c63f629 P 77cea489a5e243059d501c5c3a7c0689 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77cea489a5e243059d501c5c3a7c0689, State: Initialized, Role: FOLLOWER
I20260929 01:06:08.520658 18981 consensus_queue.cc:260] T 79ae7bc9a0184e1cb8b183b51c63f629 P 77cea489a5e243059d501c5c3a7c0689 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048ffc2857be456f85d61b8d75cf6bfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "2a35a86ffa7b45b495ec63ce2e59bdaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42597 } } peers { permanent_uuid: "77cea489a5e243059d501c5c3a7c0689" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45105 } }
I20260929 01:06:08.521255 18963 heartbeater.cc:503] Master 127.0.98.62:33389 was elected leader, sending a full tablet report...
I20260929 01:06:08.521548 18981 ts_tablet_manager.cc:1428] T 79ae7bc9a0184e1cb8b183b51c63f629 P 77cea489a5e243059d501c5c3a7c0689: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 01:06:08.614852 18986 raft_consensus.cc:482] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:06:08.615124 18986 raft_consensus.cc:504] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048ffc2857be456f85d61b8d75cf6bfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "2a35a86ffa7b45b495ec63ce2e59bdaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42597 } } peers { permanent_uuid: "77cea489a5e243059d501c5c3a7c0689" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45105 } }
I20260929 01:06:08.616325 18986 leader_election.cc:288] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 048ffc2857be456f85d61b8d75cf6bfa (127.0.98.2:46003), 77cea489a5e243059d501c5c3a7c0689 (127.0.98.3:45105)
I20260929 01:06:08.621973 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:08.616722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43997 (local address 127.0.98.2:46003)
0929 01:06:08.616863 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:08.616870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:08.617003 (+ 133us) server_negotiation.cc:407] Connection header received
0929 01:06:08.617147 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:08.617150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:08.617202 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:08.617298 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:08.617687 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.618566 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.620526 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.621231 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.621362 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:08.621435 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:08.621765 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 01:06:08.621844 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 01:06:08.621973 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:08.616646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60955 (local address 127.0.98.3:45105)
0929 01:06:08.616788 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:08.616794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:08.617066 (+ 272us) server_negotiation.cc:407] Connection header received
0929 01:06:08.617140 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:08.617143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:08.617202 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:08.617298 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:08.617687 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.618561 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.620526 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.621281 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.621384 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:08.621437 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:08.621765 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 01:06:08.621849 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 01:06:08.622001 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:08.616552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46003 (local address 127.0.98.1:43997)
0929 01:06:08.616905 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:08.616920 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:08.617037 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:08.617319 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:08.617322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:08.617340 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:08.617555 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:08.617563 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.618679 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.618683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:08.620372 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:08.620384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.621390 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.621394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:08.621575 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:08.621616 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 01:06:08.621699 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 01:06:08.621809 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":259,"spinlock_wait_cycles":36096,"thread_start_us":112,"threads_started":1}
I20260929 01:06:08.621999 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:08.616552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45105 (local address 127.0.98.1:60955)
0929 01:06:08.616959 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:08.616972 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:08.617057 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:08.617353 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:08.617356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:08.617381 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:08.617559 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:08.617564 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.618679 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.618683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:08.620372 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:08.620384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.621390 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.621394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:08.621600 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:08.621616 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 01:06:08.621699 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 01:06:08.621809 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":382,"thread_start_us":106,"threads_started":1}
I20260929 01:06:08.622720 18768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ae7bc9a0184e1cb8b183b51c63f629" candidate_uuid: "2a35a86ffa7b45b495ec63ce2e59bdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048ffc2857be456f85d61b8d75cf6bfa" is_pre_election: true
I20260929 01:06:08.622720 18917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ae7bc9a0184e1cb8b183b51c63f629" candidate_uuid: "2a35a86ffa7b45b495ec63ce2e59bdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77cea489a5e243059d501c5c3a7c0689" is_pre_election: true
I20260929 01:06:08.623070 18768 raft_consensus.cc:2454] T 79ae7bc9a0184e1cb8b183b51c63f629 P 048ffc2857be456f85d61b8d75cf6bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a35a86ffa7b45b495ec63ce2e59bdaf in term 0.
I20260929 01:06:08.623070 18917 raft_consensus.cc:2454] T 79ae7bc9a0184e1cb8b183b51c63f629 P 77cea489a5e243059d501c5c3a7c0689 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a35a86ffa7b45b495ec63ce2e59bdaf in term 0.
I20260929 01:06:08.623551 18584 leader_election.cc:302] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a35a86ffa7b45b495ec63ce2e59bdaf, 77cea489a5e243059d501c5c3a7c0689; no voters:
I20260929 01:06:08.623868 18986 raft_consensus.cc:2781] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:06:08.623955 18986 raft_consensus.cc:482] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:06:08.623997 18986 raft_consensus.cc:3037] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:08.625000 18986 raft_consensus.cc:504] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048ffc2857be456f85d61b8d75cf6bfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "2a35a86ffa7b45b495ec63ce2e59bdaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42597 } } peers { permanent_uuid: "77cea489a5e243059d501c5c3a7c0689" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45105 } }
I20260929 01:06:08.625329 18986 leader_election.cc:288] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf [CANDIDATE]: Term 1 election: Requested vote from peers 048ffc2857be456f85d61b8d75cf6bfa (127.0.98.2:46003), 77cea489a5e243059d501c5c3a7c0689 (127.0.98.3:45105)
I20260929 01:06:08.625660 18768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ae7bc9a0184e1cb8b183b51c63f629" candidate_uuid: "2a35a86ffa7b45b495ec63ce2e59bdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048ffc2857be456f85d61b8d75cf6bfa"
I20260929 01:06:08.625689 18917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ae7bc9a0184e1cb8b183b51c63f629" candidate_uuid: "2a35a86ffa7b45b495ec63ce2e59bdaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77cea489a5e243059d501c5c3a7c0689"
I20260929 01:06:08.625787 18917 raft_consensus.cc:3037] T 79ae7bc9a0184e1cb8b183b51c63f629 P 77cea489a5e243059d501c5c3a7c0689 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:08.625800 18768 raft_consensus.cc:3037] T 79ae7bc9a0184e1cb8b183b51c63f629 P 048ffc2857be456f85d61b8d75cf6bfa [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:08.626888 18917 raft_consensus.cc:2454] T 79ae7bc9a0184e1cb8b183b51c63f629 P 77cea489a5e243059d501c5c3a7c0689 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a35a86ffa7b45b495ec63ce2e59bdaf in term 1.
I20260929 01:06:08.626924 18768 raft_consensus.cc:2454] T 79ae7bc9a0184e1cb8b183b51c63f629 P 048ffc2857be456f85d61b8d75cf6bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a35a86ffa7b45b495ec63ce2e59bdaf in term 1.
I20260929 01:06:08.627166 18584 leader_election.cc:302] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a35a86ffa7b45b495ec63ce2e59bdaf, 77cea489a5e243059d501c5c3a7c0689; no voters:
I20260929 01:06:08.627341 18986 raft_consensus.cc:2781] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:08.627555 18986 raft_consensus.cc:686] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf [term 1 LEADER]: Becoming Leader. State: Replica: 2a35a86ffa7b45b495ec63ce2e59bdaf, State: Running, Role: LEADER
I20260929 01:06:08.627910 18986 consensus_queue.cc:237] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048ffc2857be456f85d61b8d75cf6bfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } } peers { permanent_uuid: "2a35a86ffa7b45b495ec63ce2e59bdaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42597 } } peers { permanent_uuid: "77cea489a5e243059d501c5c3a7c0689" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45105 } }
I20260929 01:06:08.631109 18504 catalog_manager.cc:5495] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a35a86ffa7b45b495ec63ce2e59bdaf (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2a35a86ffa7b45b495ec63ce2e59bdaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048ffc2857be456f85d61b8d75cf6bfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a35a86ffa7b45b495ec63ce2e59bdaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77cea489a5e243059d501c5c3a7c0689" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45105 } health_report { overall_health: UNKNOWN } } }
I20260929 01:06:08.649416 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:08.645818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33556 (local address 127.0.98.1:42597)
0929 01:06:08.645956 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:08.645960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:08.645976 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:06:08.646100 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:08.646102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:08.646159 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:08.646270 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:08.646521 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.647087 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.647955 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.648175 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.648285 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:08.648589 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:08.648988 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:08.649145 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:08.649241 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 01:06:08.649278 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:06:08.655267 18768 raft_consensus.cc:1261] T 79ae7bc9a0184e1cb8b183b51c63f629 P 048ffc2857be456f85d61b8d75cf6bfa [term 1 FOLLOWER]: Refusing update from remote peer 2a35a86ffa7b45b495ec63ce2e59bdaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:06:08.655265 18917 raft_consensus.cc:1261] T 79ae7bc9a0184e1cb8b183b51c63f629 P 77cea489a5e243059d501c5c3a7c0689 [term 1 FOLLOWER]: Refusing update from remote peer 2a35a86ffa7b45b495ec63ce2e59bdaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:06:08.656013 18991 consensus_queue.cc:1035] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "77cea489a5e243059d501c5c3a7c0689" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:06:08.656199 18986 consensus_queue.cc:1035] T 79ae7bc9a0184e1cb8b183b51c63f629 P 2a35a86ffa7b45b495ec63ce2e59bdaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "048ffc2857be456f85d61b8d75cf6bfa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:06:08.664139 18995 mvcc.cc:205] Tried to move back new op lower bound from 7334477695604559872 to 7334477695502675968. Current Snapshot: MvccSnapshot[applied={T|T < 7334477695604559872}]
I20260929 01:06:08.671051 18504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:06:08.671203 18504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:06:08.673385 18504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24cb8b84367e403ca375b98c5f3176af: Sending DeleteTablet for 3 replicas of tablet 79ae7bc9a0184e1cb8b183b51c63f629
I20260929 01:06:08.674180 18897 tablet_service.cc:1514] Processing DeleteTablet for tablet 79ae7bc9a0184e1cb8b183b51c63f629 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:06:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52748
I20260929 01:06:08.674233 18763 tablet_service.cc:1514] Processing DeleteTablet for tablet 79ae7bc9a0184e1cb8b183b51c63f629 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:06:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36542
I20260929 01:06:08.674690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 18566
I20260929 01:06:08.678015 19005 tablet_replica.cc:330] T 79ae7bc9a0184e1cb8b183b51c63f629 P 048ffc2857be456f85d61b8d75cf6bfa: stopping tablet replica
I20260929 01:06:08.678349 19005 raft_consensus.cc:2229] T 79ae7bc9a0184e1cb8b183b51c63f629 P 048ffc2857be456f85d61b8d75cf6bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:06:08.678670 19005 raft_consensus.cc:2258] T 79ae7bc9a0184e1cb8b183b51c63f629 P 048ffc2857be456f85d61b8d75cf6bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:06:08.679077 19006 tablet_replica.cc:330] T 79ae7bc9a0184e1cb8b183b51c63f629 P 77cea489a5e243059d501c5c3a7c0689: stopping tablet replica
I20260929 01:06:08.679476 19006 raft_consensus.cc:2229] T 79ae7bc9a0184e1cb8b183b51c63f629 P 77cea489a5e243059d501c5c3a7c0689 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:06:08.679783 19006 raft_consensus.cc:2258] T 79ae7bc9a0184e1cb8b183b51c63f629 P 77cea489a5e243059d501c5c3a7c0689 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:06:08.680416 19005 ts_tablet_manager.cc:1905] T 79ae7bc9a0184e1cb8b183b51c63f629 P 048ffc2857be456f85d61b8d75cf6bfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:06:08.681183 19006 ts_tablet_manager.cc:1905] T 79ae7bc9a0184e1cb8b183b51c63f629 P 77cea489a5e243059d501c5c3a7c0689: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 01:06:08.682413 18489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42597 (error 108)
I20260929 01:06:08.682974 19005 ts_tablet_manager.cc:1918] T 79ae7bc9a0184e1cb8b183b51c63f629 P 048ffc2857be456f85d61b8d75cf6bfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:06:08.683079 19005 log.cc:1196] T 79ae7bc9a0184e1cb8b183b51c63f629 P 048ffc2857be456f85d61b8d75cf6bfa: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-1/wal/wals/79ae7bc9a0184e1cb8b183b51c63f629
I20260929 01:06:08.683341 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 18700
I20260929 01:06:08.683457 19005 ts_tablet_manager.cc:1939] T 79ae7bc9a0184e1cb8b183b51c63f629 P 048ffc2857be456f85d61b8d75cf6bfa: Deleting consensus metadata
I20260929 01:06:08.684692 19006 ts_tablet_manager.cc:1918] T 79ae7bc9a0184e1cb8b183b51c63f629 P 77cea489a5e243059d501c5c3a7c0689: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:06:08.684784 19006 log.cc:1196] T 79ae7bc9a0184e1cb8b183b51c63f629 P 77cea489a5e243059d501c5c3a7c0689: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/79ae7bc9a0184e1cb8b183b51c63f629
I20260929 01:06:08.685105 19006 ts_tablet_manager.cc:1939] T 79ae7bc9a0184e1cb8b183b51c63f629 P 77cea489a5e243059d501c5c3a7c0689: Deleting consensus metadata
I20260929 01:06:08.685112 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:08.683644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42597 (local address 127.0.0.1:33562)
0929 01:06:08.684051 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:08.684969 (+ 918us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303}
I20260929 01:06:08.686329 18490 catalog_manager.cc:4841] TS 77cea489a5e243059d501c5c3a7c0689 (127.0.98.3:45105): tablet 79ae7bc9a0184e1cb8b183b51c63f629 (table test-table [id=06672ae6c3d6483fb1dcd41698c9db33]) successfully deleted
W20260929 01:06:08.686954 18489 catalog_manager.cc:4584] TS 2a35a86ffa7b45b495ec63ce2e59bdaf (127.0.98.1:42597): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 79ae7bc9a0184e1cb8b183b51c63f629: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42597: connect: Connection refused (error 111)
I20260929 01:06:08.690088 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 18834
I20260929 01:06:08.690900 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:08.690141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46003 (local address 127.0.0.1:36558)
0929 01:06:08.690708 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:08.690771 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472}
I20260929 01:06:08.695776 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 18472
2026-09-29T01:06:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2264 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T01:06:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:08Z Disabled control of system clock
I20260929 01:06:08.725069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40395
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 01:06:08.831549 19011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:08.831784 19011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:08.831817 19011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:08.835654 19011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:08.835703 19011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:08.835719 19011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:08.835736 19011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:08.835750 19011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:08.840379 19011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40395
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:08.841310 19011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:08.842543 19011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:08.850633 19017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:08.850625 19016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:08.850665 19019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:08.850835 19011 server_base.cc:1034] running on GCE node
I20260929 01:06:08.851428 19011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:08.852531 19011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:08.853714 19011 hybrid_clock.cc:648] HybridClock initialized: now 1790643968853704 us; error 24 us; skew 500 ppm
I20260929 01:06:08.855491 19011 webserver.cc:460] Webserver started at http://127.0.98.62:39259/ using document root <none> and password file <none>
I20260929 01:06:08.856040 19011 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:08.856127 19011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:08.856359 19011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:08.858331 19011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "b3599290426144afb3d7e6e47a6ea6b7"
format_stamp: "Formatted at 2026-09-29 01:06:08 on dist-test-slave-7zdl"
server_key: "abf1214fb5de914f414e443b3ef5e740"
server_key_iv: "ade8409d290e5fe5c43329cda9453afb"
server_key_version: "encryptionkey@0"
I20260929 01:06:08.858948 19011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "b3599290426144afb3d7e6e47a6ea6b7"
format_stamp: "Formatted at 2026-09-29 01:06:08 on dist-test-slave-7zdl"
server_key: "abf1214fb5de914f414e443b3ef5e740"
server_key_iv: "ade8409d290e5fe5c43329cda9453afb"
server_key_version: "encryptionkey@0"
I20260929 01:06:08.862401 19011 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:06:08.864704 19025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:08.865803 19011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260929 01:06:08.865934 19011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "b3599290426144afb3d7e6e47a6ea6b7"
format_stamp: "Formatted at 2026-09-29 01:06:08 on dist-test-slave-7zdl"
server_key: "abf1214fb5de914f414e443b3ef5e740"
server_key_iv: "ade8409d290e5fe5c43329cda9453afb"
server_key_version: "encryptionkey@0"
I20260929 01:06:08.866040 19011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:08.884696 19011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:08.885506 19011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:08.885689 19011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:08.892573 19011 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40395
I20260929 01:06:08.892577 19077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40395 every 8 connection(s)
I20260929 01:06:08.893456 19011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:08.896471 19078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:08.901032 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 19011
I20260929 01:06:08.901137 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:08.901427 392 external_mini_cluster.cc:1428] Setting key 81db0b659ff4bb656b646e1114dfcd6a
I20260929 01:06:08.902093 19078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3599290426144afb3d7e6e47a6ea6b7: Bootstrap starting.
I20260929 01:06:08.904726 19078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3599290426144afb3d7e6e47a6ea6b7: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:08.905565 19078 log.cc:824] T 00000000000000000000000000000000 P b3599290426144afb3d7e6e47a6ea6b7: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:08.907639 19078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3599290426144afb3d7e6e47a6ea6b7: No bootstrap required, opened a new log
I20260929 01:06:08.912060 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:08.902981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55574 (local address 127.0.98.62:40395)
0929 01:06:08.903662 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:08.903680 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:08.903723 (+ 43us) server_negotiation.cc:407] Connection header received
0929 01:06:08.904517 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:08.904540 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:08.905047 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:08.905402 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:08.905827 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.907197 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.908222 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:08.908874 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:08.909039 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:08.909867 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:08.909873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:08.909888 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:08.910423 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:08.910490 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:08.910518 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:08.910623 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:08.910788 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 01:06:08.911158 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":117,"threads_started":1}
I20260929 01:06:08.911935 19078 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3599290426144afb3d7e6e47a6ea6b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3599290426144afb3d7e6e47a6ea6b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } }
I20260929 01:06:08.912176 19078 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3599290426144afb3d7e6e47a6ea6b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:08.912222 19078 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3599290426144afb3d7e6e47a6ea6b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3599290426144afb3d7e6e47a6ea6b7, State: Initialized, Role: FOLLOWER
I20260929 01:06:08.912683 19078 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3599290426144afb3d7e6e47a6ea6b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3599290426144afb3d7e6e47a6ea6b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } }
I20260929 01:06:08.912833 19078 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3599290426144afb3d7e6e47a6ea6b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:08.912899 19078 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3599290426144afb3d7e6e47a6ea6b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:08.912999 19078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3599290426144afb3d7e6e47a6ea6b7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:08.913812 19078 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3599290426144afb3d7e6e47a6ea6b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3599290426144afb3d7e6e47a6ea6b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } }
I20260929 01:06:08.914162 19078 leader_election.cc:302] T 00000000000000000000000000000000 P b3599290426144afb3d7e6e47a6ea6b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3599290426144afb3d7e6e47a6ea6b7; no voters:
I20260929 01:06:08.914422 19078 leader_election.cc:288] T 00000000000000000000000000000000 P b3599290426144afb3d7e6e47a6ea6b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:08.914541 19083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3599290426144afb3d7e6e47a6ea6b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:08.914794 19083 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3599290426144afb3d7e6e47a6ea6b7 [term 1 LEADER]: Becoming Leader. State: Replica: b3599290426144afb3d7e6e47a6ea6b7, State: Running, Role: LEADER
I20260929 01:06:08.915159 19083 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3599290426144afb3d7e6e47a6ea6b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3599290426144afb3d7e6e47a6ea6b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } }
I20260929 01:06:08.915578 19078 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3599290426144afb3d7e6e47a6ea6b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:08.916715 19085 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3599290426144afb3d7e6e47a6ea6b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3599290426144afb3d7e6e47a6ea6b7. Latest consensus state: current_term: 1 leader_uuid: "b3599290426144afb3d7e6e47a6ea6b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3599290426144afb3d7e6e47a6ea6b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } } }
I20260929 01:06:08.916844 19085 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3599290426144afb3d7e6e47a6ea6b7 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:08.917187 19092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:08.917126 19084 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3599290426144afb3d7e6e47a6ea6b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3599290426144afb3d7e6e47a6ea6b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3599290426144afb3d7e6e47a6ea6b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } } }
I20260929 01:06:08.917315 19084 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3599290426144afb3d7e6e47a6ea6b7 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:08.920370 19092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:08.925585 19092 catalog_manager.cc:1347] Generated new cluster ID: 407a95266dce48d484f506c9e8a8850e
I20260929 01:06:08.925643 19092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:08.939761 19092 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:08.940886 19092 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:08.960470 19092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3599290426144afb3d7e6e47a6ea6b7: Generated new TSK 0
I20260929 01:06:08.961231 19092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:08.974782 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40395
--builtin_ntp_servers=127.0.98.20:36343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 01:06:09.094513 19102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:09.094743 19102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:09.094779 19102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:09.098623 19102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:09.098686 19102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:09.098753 19102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:06:09.103403 19102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:09.104285 19102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:09.105520 19102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:09.114612 19107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:09.114714 19102 server_base.cc:1034] running on GCE node
W20260929 01:06:09.114626 19108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:09.114630 19110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:09.115214 19102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:09.115914 19102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:09.117097 19102 hybrid_clock.cc:648] HybridClock initialized: now 1790643969117062 us; error 46 us; skew 500 ppm
I20260929 01:06:09.119043 19102 webserver.cc:460] Webserver started at http://127.0.98.1:34335/ using document root <none> and password file <none>
I20260929 01:06:09.119594 19102 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:09.119681 19102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:09.119902 19102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:09.121897 19102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "9d40cd64bbff4ee7b5ca33a2e8617ede"
format_stamp: "Formatted at 2026-09-29 01:06:09 on dist-test-slave-7zdl"
server_key: "5d31fda1a7c04eff080da06d586f42b8"
server_key_iv: "d850d8b79af1629930806000448126ba"
server_key_version: "encryptionkey@0"
I20260929 01:06:09.122447 19102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9d40cd64bbff4ee7b5ca33a2e8617ede"
format_stamp: "Formatted at 2026-09-29 01:06:09 on dist-test-slave-7zdl"
server_key: "5d31fda1a7c04eff080da06d586f42b8"
server_key_iv: "d850d8b79af1629930806000448126ba"
server_key_version: "encryptionkey@0"
I20260929 01:06:09.125988 19102 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.001s
I20260929 01:06:09.128402 19116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:09.129530 19102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:06:09.129659 19102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "9d40cd64bbff4ee7b5ca33a2e8617ede"
format_stamp: "Formatted at 2026-09-29 01:06:09 on dist-test-slave-7zdl"
server_key: "5d31fda1a7c04eff080da06d586f42b8"
server_key_iv: "d850d8b79af1629930806000448126ba"
server_key_version: "encryptionkey@0"
I20260929 01:06:09.129765 19102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:09.156136 19102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:09.156802 19102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:09.156998 19102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:09.157598 19102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:06:09.158529 19102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:06:09.158599 19102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:09.158666 19102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:06:09.158712 19102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:09.167248 19102 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41891
I20260929 01:06:09.167275 19229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41891 every 8 connection(s)
I20260929 01:06:09.168135 19102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:06:09.171713 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 19102
I20260929 01:06:09.171825 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:06:09.172117 392 external_mini_cluster.cc:1428] Setting key 771bd78b8dea64d522278a4772456892
I20260929 01:06:09.174139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40395
--builtin_ntp_servers=127.0.98.20:36343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 01:06:09.179088 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:09.169947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56297 (local address 127.0.98.62:40395)
0929 01:06:09.170107 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:09.170113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:09.171303 (+ 1190us) server_negotiation.cc:407] Connection header received
0929 01:06:09.172984 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:09.172987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:09.173029 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:09.173100 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:09.174132 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.175009 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.176745 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.177016 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.177144 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:09.178506 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:09.178509 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:09.178511 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:09.178522 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:09.178539 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:09.178547 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:09.178604 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:09.178869 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 01:06:09.178939 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 01:06:09.180356 19232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:09.170305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40395 (local address 127.0.98.1:56297)
0929 01:06:09.171160 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:09.171203 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:09.172771 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:09.173281 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:09.173289 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:09.173400 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:09.173980 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:09.173993 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.175169 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.175172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:09.176603 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:09.176618 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.177208 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.177215 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:09.177466 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:09.178191 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:06:09.178195 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:09.178254 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:06:09.178343 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:09.178637 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:09.178647 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:09.178656 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 01:06:09.178858 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 01:06:09.179200 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":117,"threads_started":1}
I20260929 01:06:09.181624 19230 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40395
I20260929 01:06:09.181856 19230 heartbeater.cc:464] Registering TS with master...
I20260929 01:06:09.182345 19230 heartbeater.cc:511] Master 127.0.98.62:40395 requested a full tablet report, sending...
I20260929 01:06:09.183840 19042 ts_manager.cc:194] Registered new tserver with Master: 9d40cd64bbff4ee7b5ca33a2e8617ede (127.0.98.1:41891)
I20260929 01:06:09.184978 19042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:56297
W20260929 01:06:09.284291 19233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:09.284580 19233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:09.284658 19233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:09.288646 19233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:09.288722 19233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:09.288806 19233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:06:09.293533 19233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:09.294528 19233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:09.295892 19233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:09.305388 19239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:09.305545 19233 server_base.cc:1034] running on GCE node
W20260929 01:06:09.305388 19238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:09.305454 19241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:09.306025 19233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:09.306649 19233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:09.307823 19233 hybrid_clock.cc:648] HybridClock initialized: now 1790643969307791 us; error 44 us; skew 500 ppm
I20260929 01:06:09.309553 19233 webserver.cc:460] Webserver started at http://127.0.98.2:39721/ using document root <none> and password file <none>
I20260929 01:06:09.310109 19233 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:09.310195 19233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:09.310415 19233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:09.312418 19233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "2be6f90964594354870d9e79347be480"
format_stamp: "Formatted at 2026-09-29 01:06:09 on dist-test-slave-7zdl"
server_key: "ccee41fd5af73f27cbdc1a10e65935be"
server_key_iv: "f39ba491ed354e415a95a9720f79548f"
server_key_version: "encryptionkey@0"
I20260929 01:06:09.312937 19233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2be6f90964594354870d9e79347be480"
format_stamp: "Formatted at 2026-09-29 01:06:09 on dist-test-slave-7zdl"
server_key: "ccee41fd5af73f27cbdc1a10e65935be"
server_key_iv: "f39ba491ed354e415a95a9720f79548f"
server_key_version: "encryptionkey@0"
I20260929 01:06:09.316588 19233 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 01:06:09.319068 19247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:09.320185 19233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 01:06:09.320335 19233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "2be6f90964594354870d9e79347be480"
format_stamp: "Formatted at 2026-09-29 01:06:09 on dist-test-slave-7zdl"
server_key: "ccee41fd5af73f27cbdc1a10e65935be"
server_key_iv: "f39ba491ed354e415a95a9720f79548f"
server_key_version: "encryptionkey@0"
I20260929 01:06:09.320468 19233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:09.330039 19233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:09.330660 19233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:09.330924 19233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:09.331445 19233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:06:09.332270 19233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:06:09.332335 19233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:09.332419 19233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:06:09.332460 19233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:09.340677 19233 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46281
I20260929 01:06:09.340751 19360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46281 every 8 connection(s)
I20260929 01:06:09.341588 19233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:06:09.350337 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:09.343444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56165 (local address 127.0.98.62:40395)
0929 01:06:09.343619 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:09.343623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:09.344599 (+ 976us) server_negotiation.cc:407] Connection header received
0929 01:06:09.345963 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:09.345966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:09.346011 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:09.346082 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:09.346987 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.347502 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.348633 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.348831 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.348900 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:09.349927 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:09.349930 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:09.349932 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:09.349942 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:09.349960 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:09.349968 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:09.350023 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:09.350180 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 01:06:09.350225 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 01:06:09.350987 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 19233
I20260929 01:06:09.351116 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:06:09.351264 19363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:09.343827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40395 (local address 127.0.98.2:56165)
0929 01:06:09.344459 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:09.344501 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:09.345754 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:09.346250 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:09.346256 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:09.346335 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:09.346852 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:09.346876 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.347616 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.347618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:09.348532 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:09.348542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.348961 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.348964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:09.349149 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:09.349693 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:06:09.349694 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:09.349731 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:06:09.349793 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:09.350055 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:09.350062 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:09.350066 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 01:06:09.350157 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 01:06:09.350355 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":103,"threads_started":1}
I20260929 01:06:09.351444 392 external_mini_cluster.cc:1428] Setting key e6c46bd770dd150de1f6303acc731f94
I20260929 01:06:09.352480 19361 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40395
I20260929 01:06:09.352746 19361 heartbeater.cc:464] Registering TS with master...
I20260929 01:06:09.353226 19361 heartbeater.cc:511] Master 127.0.98.62:40395 requested a full tablet report, sending...
I20260929 01:06:09.353451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40395
--builtin_ntp_servers=127.0.98.20:36343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 01:06:09.354223 19042 ts_manager.cc:194] Registered new tserver with Master: 2be6f90964594354870d9e79347be480 (127.0.98.2:46281)
I20260929 01:06:09.354923 19042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:56165
W20260929 01:06:09.462320 19364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:09.462554 19364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:09.462591 19364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:09.466564 19364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:09.466614 19364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:09.466671 19364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:06:09.471433 19364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:09.472352 19364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:09.473616 19364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:09.482957 19369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:09.483034 19370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:09.483103 19364 server_base.cc:1034] running on GCE node
W20260929 01:06:09.482957 19372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:09.483531 19364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:09.484125 19364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:09.485281 19364 hybrid_clock.cc:648] HybridClock initialized: now 1790643969485269 us; error 23 us; skew 500 ppm
I20260929 01:06:09.487056 19364 webserver.cc:460] Webserver started at http://127.0.98.3:38455/ using document root <none> and password file <none>
I20260929 01:06:09.487596 19364 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:09.487660 19364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:09.487896 19364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:09.489883 19364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "130b3cd26c954420be35959c6a6c034d"
format_stamp: "Formatted at 2026-09-29 01:06:09 on dist-test-slave-7zdl"
server_key: "25a2f63301b53cf851636d08a821b264"
server_key_iv: "e1b381c01a53d0c566bac8463412dc88"
server_key_version: "encryptionkey@0"
I20260929 01:06:09.490422 19364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "130b3cd26c954420be35959c6a6c034d"
format_stamp: "Formatted at 2026-09-29 01:06:09 on dist-test-slave-7zdl"
server_key: "25a2f63301b53cf851636d08a821b264"
server_key_iv: "e1b381c01a53d0c566bac8463412dc88"
server_key_version: "encryptionkey@0"
I20260929 01:06:09.493886 19364 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 01:06:09.496208 19378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:09.497349 19364 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:06:09.497483 19364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "130b3cd26c954420be35959c6a6c034d"
format_stamp: "Formatted at 2026-09-29 01:06:09 on dist-test-slave-7zdl"
server_key: "25a2f63301b53cf851636d08a821b264"
server_key_iv: "e1b381c01a53d0c566bac8463412dc88"
server_key_version: "encryptionkey@0"
I20260929 01:06:09.497588 19364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:09.509940 19364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:09.510514 19364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:09.510699 19364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:09.511250 19364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:06:09.512117 19364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:06:09.512187 19364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:09.512256 19364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:06:09.512298 19364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:09.521658 19364 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35981
I20260929 01:06:09.521678 19491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35981 every 8 connection(s)
I20260929 01:06:09.522496 19364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:06:09.529611 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 19364
I20260929 01:06:09.529737 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:06:09.530037 392 external_mini_cluster.cc:1428] Setting key 0f88dc192b9f16d27b494722820b984e
I20260929 01:06:09.531462 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:09.524499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38883 (local address 127.0.98.62:40395)
0929 01:06:09.524637 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:09.524641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:09.525621 (+ 980us) server_negotiation.cc:407] Connection header received
0929 01:06:09.526928 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:09.526931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:09.526976 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:09.527050 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:09.527854 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.528373 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.529401 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.529604 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.529676 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:09.530941 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:09.530944 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:09.530947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:09.530960 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:09.530984 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:09.530993 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:09.531085 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:09.531246 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 01:06:09.531303 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 01:06:09.532374 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:09.524836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40395 (local address 127.0.98.3:38883)
0929 01:06:09.525481 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:09.525518 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:09.526731 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:09.527214 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:09.527220 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:09.527299 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:09.527738 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:09.527749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.528467 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.528469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:09.529286 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:09.529294 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.529748 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.529754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:09.529946 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:09.530644 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:06:09.530646 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:09.530706 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:06:09.530781 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:09.531096 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:09.531104 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:09.531110 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 01:06:09.531214 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 01:06:09.531476 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":127,"threads_started":1}
I20260929 01:06:09.533360 19492 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40395
I20260929 01:06:09.533578 19492 heartbeater.cc:464] Registering TS with master...
I20260929 01:06:09.534180 19492 heartbeater.cc:511] Master 127.0.98.62:40395 requested a full tablet report, sending...
I20260929 01:06:09.535141 19042 ts_manager.cc:194] Registered new tserver with Master: 130b3cd26c954420be35959c6a6c034d (127.0.98.3:35981)
I20260929 01:06:09.535638 19042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:38883
I20260929 01:06:09.543761 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:06:09.549189 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:09.545652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55576 (local address 127.0.98.62:40395)
0929 01:06:09.545793 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:09.545797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:09.545954 (+ 157us) server_negotiation.cc:407] Connection header received
0929 01:06:09.546064 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:09.546067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:09.546103 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:09.546170 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:09.546397 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.546941 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.547796 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.548039 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.548121 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:09.548837 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:09.548840 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:09.548842 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:09.548853 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:09.548870 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:09.548877 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:09.549021 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:09.549063 (+ 42us) server_negotiation.cc:299] Negotiation successful
0929 01:06:09.549092 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 01:06:09.551728 19042 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:06:09.554674 19042 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:06:09.568751 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:09.562831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41891 (local address 127.0.0.1:56682)
0929 01:06:09.563824 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:09.563838 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:09.563909 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:09.564692 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:09.564695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:09.564710 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:09.564793 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:09.564799 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.565815 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.565821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:09.567262 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:09.567272 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.568204 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.568208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:09.568317 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:09.568354 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 01:06:09.568458 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 01:06:09.568580 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"mutex_wait_us":105,"thread_start_us":120,"threads_started":1}
I20260929 01:06:09.568773 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:09.563442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56682 (local address 127.0.98.1:41891)
0929 01:06:09.564219 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:09.564231 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:09.564255 (+ 24us) server_negotiation.cc:407] Connection header received
0929 01:06:09.564334 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:09.564340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:09.564550 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:09.564716 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:09.564880 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.565702 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.567395 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.568074 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.568208 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:09.568432 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:09.568515 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 01:06:09.568613 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":96,"threads_started":1}
I20260929 01:06:09.569263 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:09.562584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46281 (local address 127.0.0.1:57906)
0929 01:06:09.562991 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:09.563043 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:09.563168 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:09.564067 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:09.564074 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:09.564097 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:09.564222 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:09.564230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.565849 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.565852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:09.567500 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:09.567510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.568750 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.568754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:09.568923 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:09.568981 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 01:06:09.569050 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 01:06:09.569124 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":116,"thread_start_us":73,"threads_started":1}
I20260929 01:06:09.569273 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:09.563452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35981 (local address 127.0.0.1:32908)
0929 01:06:09.564430 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:09.564442 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:09.564526 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:09.564960 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:09.564963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:09.564988 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:09.565059 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:09.565064 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.566461 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.566465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:09.567542 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:09.567548 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.568750 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.568754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:09.569047 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:09.569067 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 01:06:09.569126 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 01:06:09.569173 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"mutex_wait_us":20,"thread_start_us":110,"threads_started":1}
I20260929 01:06:09.569698 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:09.563942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32908 (local address 127.0.98.3:35981)
0929 01:06:09.564402 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:09.564414 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:09.564527 (+ 113us) server_negotiation.cc:407] Connection header received
0929 01:06:09.564631 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:09.564636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:09.564819 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:09.564976 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:09.565174 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.566337 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.567679 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.568420 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.568530 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:09.568686 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:09.569430 (+ 744us) server_negotiation.cc:299] Negotiation successful
0929 01:06:09.569551 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":151,"threads_started":1}
I20260929 01:06:09.569850 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:09.562760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57906 (local address 127.0.98.2:46281)
0929 01:06:09.563340 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:09.563354 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:09.563381 (+ 27us) server_negotiation.cc:407] Connection header received
0929 01:06:09.563460 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:09.563466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:09.563690 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:09.563857 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:09.564339 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.565703 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.567630 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.568420 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.568526 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:09.568686 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:09.569592 (+ 906us) server_negotiation.cc:299] Negotiation successful
0929 01:06:09.569688 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":159,"threads_started":1}
I20260929 01:06:09.572299 19164 tablet_service.cc:1467] Processing CreateTablet for tablet 780b8f01667b4c19a4ed1ed849d21523 (DEFAULT_TABLE table=test-table [id=8574cee3bf3941d9bd390788161b382e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:06:09.572647 19295 tablet_service.cc:1467] Processing CreateTablet for tablet 780b8f01667b4c19a4ed1ed849d21523 (DEFAULT_TABLE table=test-table [id=8574cee3bf3941d9bd390788161b382e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:06:09.572925 19426 tablet_service.cc:1467] Processing CreateTablet for tablet 780b8f01667b4c19a4ed1ed849d21523 (DEFAULT_TABLE table=test-table [id=8574cee3bf3941d9bd390788161b382e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:06:09.573518 19164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 780b8f01667b4c19a4ed1ed849d21523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:09.573581 19295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 780b8f01667b4c19a4ed1ed849d21523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:09.573900 19426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 780b8f01667b4c19a4ed1ed849d21523. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:09.579552 19509 tablet_bootstrap.cc:492] T 780b8f01667b4c19a4ed1ed849d21523 P 2be6f90964594354870d9e79347be480: Bootstrap starting.
I20260929 01:06:09.581115 19511 tablet_bootstrap.cc:492] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d: Bootstrap starting.
I20260929 01:06:09.581866 19510 tablet_bootstrap.cc:492] T 780b8f01667b4c19a4ed1ed849d21523 P 9d40cd64bbff4ee7b5ca33a2e8617ede: Bootstrap starting.
I20260929 01:06:09.582029 19509 tablet_bootstrap.cc:654] T 780b8f01667b4c19a4ed1ed849d21523 P 2be6f90964594354870d9e79347be480: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:09.582916 19509 log.cc:824] T 780b8f01667b4c19a4ed1ed849d21523 P 2be6f90964594354870d9e79347be480: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:09.583251 19511 tablet_bootstrap.cc:654] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:09.583891 19510 tablet_bootstrap.cc:654] T 780b8f01667b4c19a4ed1ed849d21523 P 9d40cd64bbff4ee7b5ca33a2e8617ede: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:09.584046 19511 log.cc:824] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:09.584705 19510 log.cc:824] T 780b8f01667b4c19a4ed1ed849d21523 P 9d40cd64bbff4ee7b5ca33a2e8617ede: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:09.585191 19509 tablet_bootstrap.cc:492] T 780b8f01667b4c19a4ed1ed849d21523 P 2be6f90964594354870d9e79347be480: No bootstrap required, opened a new log
I20260929 01:06:09.585405 19509 ts_tablet_manager.cc:1397] T 780b8f01667b4c19a4ed1ed849d21523 P 2be6f90964594354870d9e79347be480: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 01:06:09.586184 19511 tablet_bootstrap.cc:492] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d: No bootstrap required, opened a new log
I20260929 01:06:09.586346 19511 ts_tablet_manager.cc:1397] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 01:06:09.586621 19510 tablet_bootstrap.cc:492] T 780b8f01667b4c19a4ed1ed849d21523 P 9d40cd64bbff4ee7b5ca33a2e8617ede: No bootstrap required, opened a new log
I20260929 01:06:09.586826 19510 ts_tablet_manager.cc:1397] T 780b8f01667b4c19a4ed1ed849d21523 P 9d40cd64bbff4ee7b5ca33a2e8617ede: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 01:06:09.588863 19509 raft_consensus.cc:348] T 780b8f01667b4c19a4ed1ed849d21523 P 2be6f90964594354870d9e79347be480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be6f90964594354870d9e79347be480" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46281 } } peers { permanent_uuid: "9d40cd64bbff4ee7b5ca33a2e8617ede" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41891 } } peers { permanent_uuid: "130b3cd26c954420be35959c6a6c034d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35981 } }
I20260929 01:06:09.589126 19509 raft_consensus.cc:374] T 780b8f01667b4c19a4ed1ed849d21523 P 2be6f90964594354870d9e79347be480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:09.589192 19509 raft_consensus.cc:729] T 780b8f01667b4c19a4ed1ed849d21523 P 2be6f90964594354870d9e79347be480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2be6f90964594354870d9e79347be480, State: Initialized, Role: FOLLOWER
I20260929 01:06:09.589679 19509 consensus_queue.cc:260] T 780b8f01667b4c19a4ed1ed849d21523 P 2be6f90964594354870d9e79347be480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be6f90964594354870d9e79347be480" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46281 } } peers { permanent_uuid: "9d40cd64bbff4ee7b5ca33a2e8617ede" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41891 } } peers { permanent_uuid: "130b3cd26c954420be35959c6a6c034d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35981 } }
I20260929 01:06:09.590354 19361 heartbeater.cc:503] Master 127.0.98.62:40395 was elected leader, sending a full tablet report...
I20260929 01:06:09.590471 19511 raft_consensus.cc:348] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be6f90964594354870d9e79347be480" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46281 } } peers { permanent_uuid: "9d40cd64bbff4ee7b5ca33a2e8617ede" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41891 } } peers { permanent_uuid: "130b3cd26c954420be35959c6a6c034d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35981 } }
I20260929 01:06:09.590695 19511 raft_consensus.cc:374] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:09.590746 19511 raft_consensus.cc:729] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 130b3cd26c954420be35959c6a6c034d, State: Initialized, Role: FOLLOWER
I20260929 01:06:09.590742 19509 ts_tablet_manager.cc:1428] T 780b8f01667b4c19a4ed1ed849d21523 P 2be6f90964594354870d9e79347be480: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 01:06:09.591055 19510 raft_consensus.cc:348] T 780b8f01667b4c19a4ed1ed849d21523 P 9d40cd64bbff4ee7b5ca33a2e8617ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be6f90964594354870d9e79347be480" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46281 } } peers { permanent_uuid: "9d40cd64bbff4ee7b5ca33a2e8617ede" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41891 } } peers { permanent_uuid: "130b3cd26c954420be35959c6a6c034d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35981 } }
I20260929 01:06:09.591284 19510 raft_consensus.cc:374] T 780b8f01667b4c19a4ed1ed849d21523 P 9d40cd64bbff4ee7b5ca33a2e8617ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:09.591331 19510 raft_consensus.cc:729] T 780b8f01667b4c19a4ed1ed849d21523 P 9d40cd64bbff4ee7b5ca33a2e8617ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d40cd64bbff4ee7b5ca33a2e8617ede, State: Initialized, Role: FOLLOWER
I20260929 01:06:09.591341 19511 consensus_queue.cc:260] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be6f90964594354870d9e79347be480" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46281 } } peers { permanent_uuid: "9d40cd64bbff4ee7b5ca33a2e8617ede" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41891 } } peers { permanent_uuid: "130b3cd26c954420be35959c6a6c034d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35981 } }
I20260929 01:06:09.591709 19510 consensus_queue.cc:260] T 780b8f01667b4c19a4ed1ed849d21523 P 9d40cd64bbff4ee7b5ca33a2e8617ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be6f90964594354870d9e79347be480" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46281 } } peers { permanent_uuid: "9d40cd64bbff4ee7b5ca33a2e8617ede" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41891 } } peers { permanent_uuid: "130b3cd26c954420be35959c6a6c034d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35981 } }
I20260929 01:06:09.592192 19492 heartbeater.cc:503] Master 127.0.98.62:40395 was elected leader, sending a full tablet report...
I20260929 01:06:09.592280 19511 ts_tablet_manager.cc:1428] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260929 01:06:09.592432 19362 tablet.cc:2369] T 780b8f01667b4c19a4ed1ed849d21523 P 2be6f90964594354870d9e79347be480: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:06:09.592983 19510 ts_tablet_manager.cc:1428] T 780b8f01667b4c19a4ed1ed849d21523 P 9d40cd64bbff4ee7b5ca33a2e8617ede: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 01:06:09.593262 19230 heartbeater.cc:503] Master 127.0.98.62:40395 was elected leader, sending a full tablet report...
I20260929 01:06:09.597777 19516 raft_consensus.cc:482] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:06:09.597909 19516 raft_consensus.cc:504] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be6f90964594354870d9e79347be480" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46281 } } peers { permanent_uuid: "9d40cd64bbff4ee7b5ca33a2e8617ede" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41891 } } peers { permanent_uuid: "130b3cd26c954420be35959c6a6c034d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35981 } }
I20260929 01:06:09.598932 19516 leader_election.cc:288] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2be6f90964594354870d9e79347be480 (127.0.98.2:46281), 9d40cd64bbff4ee7b5ca33a2e8617ede (127.0.98.1:41891)
I20260929 01:06:09.603240 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:09.599154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46281 (local address 127.0.98.3:37061)
0929 01:06:09.599380 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:09.599399 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:09.599504 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:09.599823 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:09.599826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:09.599840 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:09.599958 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:09.599965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.600870 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.600874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:09.601814 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:09.601824 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.602721 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.602725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:09.602849 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:09.602875 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 01:06:09.602960 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 01:06:09.603095 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":1}
I20260929 01:06:09.603241 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:09.599223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37061 (local address 127.0.98.2:46281)
0929 01:06:09.599385 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:09.599390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:09.599608 (+ 218us) server_negotiation.cc:407] Connection header received
0929 01:06:09.599661 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:09.599665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:09.599722 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:09.599804 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:09.600078 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.600761 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.601934 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.602586 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.602680 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:09.602744 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:09.602982 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 01:06:09.603114 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 01:06:09.603668 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:09.599154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41891 (local address 127.0.98.3:60249)
0929 01:06:09.599562 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:09.599577 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:09.599668 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:09.600018 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:09.600024 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:09.600044 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:09.600164 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:09.600170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.601140 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.601143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:09.602268 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:09.602278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.603089 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.603093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:09.603197 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:09.603221 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 01:06:09.603310 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 01:06:09.603454 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":84,"thread_start_us":104,"threads_started":1}
I20260929 01:06:09.603797 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:09.599315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60249 (local address 127.0.98.1:41891)
0929 01:06:09.599561 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:09.599566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:09.599658 (+ 92us) server_negotiation.cc:407] Connection header received
0929 01:06:09.599765 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:09.599768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:09.599822 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:09.599901 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:09.600296 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.600890 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.602399 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.602971 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.603074 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:09.603130 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:09.603600 (+ 470us) server_negotiation.cc:299] Negotiation successful
0929 01:06:09.603696 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260929 01:06:09.603886 19315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "780b8f01667b4c19a4ed1ed849d21523" candidate_uuid: "130b3cd26c954420be35959c6a6c034d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2be6f90964594354870d9e79347be480" is_pre_election: true
I20260929 01:06:09.604166 19315 raft_consensus.cc:2454] T 780b8f01667b4c19a4ed1ed849d21523 P 2be6f90964594354870d9e79347be480 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 130b3cd26c954420be35959c6a6c034d in term 0.
I20260929 01:06:09.604264 19184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "780b8f01667b4c19a4ed1ed849d21523" candidate_uuid: "130b3cd26c954420be35959c6a6c034d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d40cd64bbff4ee7b5ca33a2e8617ede" is_pre_election: true
I20260929 01:06:09.604521 19184 raft_consensus.cc:2454] T 780b8f01667b4c19a4ed1ed849d21523 P 9d40cd64bbff4ee7b5ca33a2e8617ede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 130b3cd26c954420be35959c6a6c034d in term 0.
I20260929 01:06:09.604624 19379 leader_election.cc:302] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 130b3cd26c954420be35959c6a6c034d, 2be6f90964594354870d9e79347be480; no voters:
I20260929 01:06:09.604875 19516 raft_consensus.cc:2781] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:06:09.604969 19516 raft_consensus.cc:482] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:06:09.605015 19516 raft_consensus.cc:3037] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:09.605983 19516 raft_consensus.cc:504] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be6f90964594354870d9e79347be480" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46281 } } peers { permanent_uuid: "9d40cd64bbff4ee7b5ca33a2e8617ede" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41891 } } peers { permanent_uuid: "130b3cd26c954420be35959c6a6c034d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35981 } }
I20260929 01:06:09.606287 19516 leader_election.cc:288] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d [CANDIDATE]: Term 1 election: Requested vote from peers 2be6f90964594354870d9e79347be480 (127.0.98.2:46281), 9d40cd64bbff4ee7b5ca33a2e8617ede (127.0.98.1:41891)
I20260929 01:06:09.606528 19315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "780b8f01667b4c19a4ed1ed849d21523" candidate_uuid: "130b3cd26c954420be35959c6a6c034d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2be6f90964594354870d9e79347be480"
I20260929 01:06:09.606613 19184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "780b8f01667b4c19a4ed1ed849d21523" candidate_uuid: "130b3cd26c954420be35959c6a6c034d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d40cd64bbff4ee7b5ca33a2e8617ede"
I20260929 01:06:09.606653 19315 raft_consensus.cc:3037] T 780b8f01667b4c19a4ed1ed849d21523 P 2be6f90964594354870d9e79347be480 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:09.606714 19184 raft_consensus.cc:3037] T 780b8f01667b4c19a4ed1ed849d21523 P 9d40cd64bbff4ee7b5ca33a2e8617ede [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:09.607905 19184 raft_consensus.cc:2454] T 780b8f01667b4c19a4ed1ed849d21523 P 9d40cd64bbff4ee7b5ca33a2e8617ede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 130b3cd26c954420be35959c6a6c034d in term 1.
I20260929 01:06:09.607904 19315 raft_consensus.cc:2454] T 780b8f01667b4c19a4ed1ed849d21523 P 2be6f90964594354870d9e79347be480 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 130b3cd26c954420be35959c6a6c034d in term 1.
I20260929 01:06:09.608286 19382 leader_election.cc:302] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 130b3cd26c954420be35959c6a6c034d, 2be6f90964594354870d9e79347be480, 9d40cd64bbff4ee7b5ca33a2e8617ede; no voters:
I20260929 01:06:09.608476 19516 raft_consensus.cc:2781] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:09.608696 19516 raft_consensus.cc:686] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d [term 1 LEADER]: Becoming Leader. State: Replica: 130b3cd26c954420be35959c6a6c034d, State: Running, Role: LEADER
I20260929 01:06:09.608981 19516 consensus_queue.cc:237] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be6f90964594354870d9e79347be480" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46281 } } peers { permanent_uuid: "9d40cd64bbff4ee7b5ca33a2e8617ede" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41891 } } peers { permanent_uuid: "130b3cd26c954420be35959c6a6c034d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35981 } }
I20260929 01:06:09.611950 19041 catalog_manager.cc:5495] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d reported cstate change: term changed from 0 to 1, leader changed from <none> to 130b3cd26c954420be35959c6a6c034d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "130b3cd26c954420be35959c6a6c034d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be6f90964594354870d9e79347be480" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d40cd64bbff4ee7b5ca33a2e8617ede" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "130b3cd26c954420be35959c6a6c034d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35981 } health_report { overall_health: HEALTHY } } }
I20260929 01:06:09.631523 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:09.627892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32918 (local address 127.0.98.3:35981)
0929 01:06:09.628027 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:09.628031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:09.628045 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:06:09.628192 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:09.628195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:09.628249 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:09.628341 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:09.628548 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.629112 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.629913 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:09.630163 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:09.630247 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:09.630494 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:09.630977 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:09.631100 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:09.631273 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 01:06:09.631314 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 01:06:09.637328 19315 raft_consensus.cc:1261] T 780b8f01667b4c19a4ed1ed849d21523 P 2be6f90964594354870d9e79347be480 [term 1 FOLLOWER]: Refusing update from remote peer 130b3cd26c954420be35959c6a6c034d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:06:09.637356 19184 raft_consensus.cc:1261] T 780b8f01667b4c19a4ed1ed849d21523 P 9d40cd64bbff4ee7b5ca33a2e8617ede [term 1 FOLLOWER]: Refusing update from remote peer 130b3cd26c954420be35959c6a6c034d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:06:09.638043 19516 consensus_queue.cc:1035] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be6f90964594354870d9e79347be480" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:06:09.638289 19519 consensus_queue.cc:1035] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d40cd64bbff4ee7b5ca33a2e8617ede" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:06:09.654222 19040 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:06:09.654414 19040 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:06:09.657066 19040 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3599290426144afb3d7e6e47a6ea6b7: Sending DeleteTablet for 3 replicas of tablet 780b8f01667b4c19a4ed1ed849d21523
I20260929 01:06:09.657833 19295 tablet_service.cc:1514] Processing DeleteTablet for tablet 780b8f01667b4c19a4ed1ed849d21523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:06:09 UTC) from {username='slave'} at 127.0.0.1:57906
I20260929 01:06:09.657997 19164 tablet_service.cc:1514] Processing DeleteTablet for tablet 780b8f01667b4c19a4ed1ed849d21523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:06:09 UTC) from {username='slave'} at 127.0.0.1:56682
I20260929 01:06:09.658381 19532 tablet_replica.cc:330] T 780b8f01667b4c19a4ed1ed849d21523 P 2be6f90964594354870d9e79347be480: stopping tablet replica
I20260929 01:06:09.658439 19533 tablet_replica.cc:330] T 780b8f01667b4c19a4ed1ed849d21523 P 9d40cd64bbff4ee7b5ca33a2e8617ede: stopping tablet replica
I20260929 01:06:09.658583 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 19102
I20260929 01:06:09.658555 19426 tablet_service.cc:1514] Processing DeleteTablet for tablet 780b8f01667b4c19a4ed1ed849d21523 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:06:09 UTC) from {username='slave'} at 127.0.0.1:32908
I20260929 01:06:09.658779 19533 raft_consensus.cc:2229] T 780b8f01667b4c19a4ed1ed849d21523 P 9d40cd64bbff4ee7b5ca33a2e8617ede [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:06:09.660996 19532 raft_consensus.cc:2229] T 780b8f01667b4c19a4ed1ed849d21523 P 2be6f90964594354870d9e79347be480 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:06:09.661338 19532 raft_consensus.cc:2258] T 780b8f01667b4c19a4ed1ed849d21523 P 2be6f90964594354870d9e79347be480 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:06:09.663002 19532 ts_tablet_manager.cc:1905] T 780b8f01667b4c19a4ed1ed849d21523 P 2be6f90964594354870d9e79347be480: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 01:06:09.665297 19029 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41891 (error 108)
I20260929 01:06:09.665805 19532 ts_tablet_manager.cc:1918] T 780b8f01667b4c19a4ed1ed849d21523 P 2be6f90964594354870d9e79347be480: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:06:09.665906 19532 log.cc:1196] T 780b8f01667b4c19a4ed1ed849d21523 P 2be6f90964594354870d9e79347be480: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-1/wal/wals/780b8f01667b4c19a4ed1ed849d21523
I20260929 01:06:09.666234 19532 ts_tablet_manager.cc:1939] T 780b8f01667b4c19a4ed1ed849d21523 P 2be6f90964594354870d9e79347be480: Deleting consensus metadata
I20260929 01:06:09.666501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 19233
I20260929 01:06:09.666541 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:09.666195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41891 (local address 127.0.0.1:56692)
0929 01:06:09.666382 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:09.666453 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 01:06:09.667155 19534 tablet_replica.cc:330] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d: stopping tablet replica
I20260929 01:06:09.667398 19534 raft_consensus.cc:2229] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:06:09.667713 19534 raft_consensus.cc:2258] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 01:06:09.668740 19029 catalog_manager.cc:4584] TS 9d40cd64bbff4ee7b5ca33a2e8617ede (127.0.98.1:41891): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 780b8f01667b4c19a4ed1ed849d21523: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41891: connect: Connection refused (error 111)
I20260929 01:06:09.669071 19534 ts_tablet_manager.cc:1905] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:06:09.671695 19534 ts_tablet_manager.cc:1918] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:06:09.671792 19534 log.cc:1196] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/780b8f01667b4c19a4ed1ed849d21523
I20260929 01:06:09.672099 19534 ts_tablet_manager.cc:1939] T 780b8f01667b4c19a4ed1ed849d21523 P 130b3cd26c954420be35959c6a6c034d: Deleting consensus metadata
I20260929 01:06:09.672871 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 19364
I20260929 01:06:09.673736 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:09.673258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46281 (local address 127.0.0.1:57922)
0929 01:06:09.673575 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:09.673638 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212}
I20260929 01:06:09.678893 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 19011
I20260929 01:06:09.679152 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:09.678922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35981 (local address 127.0.0.1:32934)
0929 01:06:09.679037 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:09.679082 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:35981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
2026-09-29T01:06:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (982 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:09 dist-test-slave-7zdl krb5kdc[19538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:09 dist-test-slave-7zdl krb5kdc[19538](info): set up 2 sockets
Sep 29 01:06:09 dist-test-slave-7zdl krb5kdc[19538](info): commencing operation
krb5kdc: starting...
W20260929 01:06:10.726866 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:06:10 dist-test-slave-7zdl krb5kdc[19538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:06:10.846458 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34705
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:10.960922 19551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:10.961232 19551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:10.961308 19551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:10.965293 19551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:10.965395 19551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:10.965432 19551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:10.965490 19551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:10.965533 19551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:10.970520 19551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:10.971555 19551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:10.972532 19551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:10.980895 19559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:10.980872 19551 server_base.cc:1034] running on GCE node
W20260929 01:06:10.980891 19556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:10.980854 19557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:10.981853 19551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:10.983151 19551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:10.984360 19551 hybrid_clock.cc:648] HybridClock initialized: now 1790643970984342 us; error 31 us; skew 500 ppm
Sep 29 01:06:10 dist-test-slave-7zdl krb5kdc[19538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:10.987183 19551 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:06:10.988049 19551 webserver.cc:460] Webserver started at http://127.0.98.62:46623/ using document root <none> and password file <none>
I20260929 01:06:10.988591 19551 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:10.988685 19551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:10.988910 19551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:10.990969 19551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "589c66e4515e4fc0bc396d103d30d8db"
format_stamp: "Formatted at 2026-09-29 01:06:10 on dist-test-slave-7zdl"
server_key: "39257599771b4bbeca6637efd02f01b6"
server_key_iv: "de149a4ebf8790a58838d0a1e57461dc"
server_key_version: "encryptionkey@0"
I20260929 01:06:10.991516 19551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "589c66e4515e4fc0bc396d103d30d8db"
format_stamp: "Formatted at 2026-09-29 01:06:10 on dist-test-slave-7zdl"
server_key: "39257599771b4bbeca6637efd02f01b6"
server_key_iv: "de149a4ebf8790a58838d0a1e57461dc"
server_key_version: "encryptionkey@0"
I20260929 01:06:10.995057 19551 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 01:06:10.997327 19566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:10.998546 19551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:06:10.998714 19551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "589c66e4515e4fc0bc396d103d30d8db"
format_stamp: "Formatted at 2026-09-29 01:06:10 on dist-test-slave-7zdl"
server_key: "39257599771b4bbeca6637efd02f01b6"
server_key_iv: "de149a4ebf8790a58838d0a1e57461dc"
server_key_version: "encryptionkey@0"
I20260929 01:06:10.998845 19551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:11.019249 19551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:11.022503 19551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:11.022724 19551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:11.029755 19551 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34705
I20260929 01:06:11.029771 19618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34705 every 8 connection(s)
I20260929 01:06:11.030681 19551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:11.031996 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 19551
I20260929 01:06:11.032141 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:11.032440 392 external_mini_cluster.cc:1428] Setting key 130f5fb35d316194e04c1dc5fa052b9c
I20260929 01:06:11.034024 19619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:11.039597 19619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 589c66e4515e4fc0bc396d103d30d8db: Bootstrap starting.
Sep 29 01:06:11 dist-test-slave-7zdl krb5kdc[19538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:06:11.041932 19619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 589c66e4515e4fc0bc396d103d30d8db: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:11.042775 19619 log.cc:824] T 00000000000000000000000000000000 P 589c66e4515e4fc0bc396d103d30d8db: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:11.045003 19619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 589c66e4515e4fc0bc396d103d30d8db: No bootstrap required, opened a new log
I20260929 01:06:11.046253 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:11.033823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35598 (local address 127.0.98.62:34705)
0929 01:06:11.034288 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:11.034303 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:11.034337 (+ 34us) server_negotiation.cc:407] Connection header received
0929 01:06:11.035020 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:11.035038 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:11.035360 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:11.035691 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:11.036482 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.037470 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.038332 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.038857 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.039025 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:11.041497 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:11.041502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:11.041515 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:11.041541 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:11.043418 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:11.043873 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:11.043878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:11.043884 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:11.044013 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:11.044268 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:11.044271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:11.044273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:11.044745 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:11.044929 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:11.045252 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 01:06:11.045552 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":142,"threads_started":1}
I20260929 01:06:11.048614 19619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 589c66e4515e4fc0bc396d103d30d8db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589c66e4515e4fc0bc396d103d30d8db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34705 } }
I20260929 01:06:11.048857 19619 raft_consensus.cc:374] T 00000000000000000000000000000000 P 589c66e4515e4fc0bc396d103d30d8db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:11.048934 19619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 589c66e4515e4fc0bc396d103d30d8db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 589c66e4515e4fc0bc396d103d30d8db, State: Initialized, Role: FOLLOWER
I20260929 01:06:11.049383 19619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 589c66e4515e4fc0bc396d103d30d8db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589c66e4515e4fc0bc396d103d30d8db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34705 } }
I20260929 01:06:11.049530 19619 raft_consensus.cc:388] T 00000000000000000000000000000000 P 589c66e4515e4fc0bc396d103d30d8db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:11.049619 19619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 589c66e4515e4fc0bc396d103d30d8db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:11.049752 19619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 589c66e4515e4fc0bc396d103d30d8db [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:11.050647 19619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 589c66e4515e4fc0bc396d103d30d8db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589c66e4515e4fc0bc396d103d30d8db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34705 } }
I20260929 01:06:11.051015 19619 leader_election.cc:302] T 00000000000000000000000000000000 P 589c66e4515e4fc0bc396d103d30d8db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 589c66e4515e4fc0bc396d103d30d8db; no voters:
I20260929 01:06:11.051324 19619 leader_election.cc:288] T 00000000000000000000000000000000 P 589c66e4515e4fc0bc396d103d30d8db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:11.051463 19624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 589c66e4515e4fc0bc396d103d30d8db [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:11.051715 19624 raft_consensus.cc:686] T 00000000000000000000000000000000 P 589c66e4515e4fc0bc396d103d30d8db [term 1 LEADER]: Becoming Leader. State: Replica: 589c66e4515e4fc0bc396d103d30d8db, State: Running, Role: LEADER
I20260929 01:06:11.051995 19624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 589c66e4515e4fc0bc396d103d30d8db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589c66e4515e4fc0bc396d103d30d8db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34705 } }
I20260929 01:06:11.052564 19619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 589c66e4515e4fc0bc396d103d30d8db [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:11.053642 19626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 589c66e4515e4fc0bc396d103d30d8db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 589c66e4515e4fc0bc396d103d30d8db. Latest consensus state: current_term: 1 leader_uuid: "589c66e4515e4fc0bc396d103d30d8db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589c66e4515e4fc0bc396d103d30d8db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34705 } } }
I20260929 01:06:11.053694 19625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 589c66e4515e4fc0bc396d103d30d8db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "589c66e4515e4fc0bc396d103d30d8db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589c66e4515e4fc0bc396d103d30d8db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34705 } } }
I20260929 01:06:11.053771 19626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 589c66e4515e4fc0bc396d103d30d8db [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:11.054011 19625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 589c66e4515e4fc0bc396d103d30d8db [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:11.054205 19633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:11.057021 19633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:11.062897 19633 catalog_manager.cc:1347] Generated new cluster ID: d75d03e37a154cf3b9ef4b085accacd3
I20260929 01:06:11.062965 19633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:11.075105 19633 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:11.075953 19633 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:11.081714 19633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 589c66e4515e4fc0bc396d103d30d8db: Generated new TSK 0
I20260929 01:06:11.082337 19633 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 01:06:11.117578 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34705
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:11.226563 19645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:11.226861 19645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:11.226917 19645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:11.230777 19645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:11.230839 19645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:11.230895 19645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:06:11.236002 19645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:11.236902 19645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:11.237828 19645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:06:11.247025 19645 server_base.cc:1034] running on GCE node
W20260929 01:06:11.247129 19651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:11.247205 19650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:11.246991 19653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:11.247615 19645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:11.248236 19645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:11.249390 19645 hybrid_clock.cc:648] HybridClock initialized: now 1790643971249367 us; error 37 us; skew 500 ppm
Sep 29 01:06:11 dist-test-slave-7zdl krb5kdc[19538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:11.252007 19645 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 01:06:11.252882 19645 webserver.cc:460] Webserver started at http://127.0.98.1:42165/ using document root <none> and password file <none>
I20260929 01:06:11.253436 19645 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:11.253523 19645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:11.253765 19645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:11.255791 19645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "fb1f52d08d8e4a63bc7a702595b68d44"
format_stamp: "Formatted at 2026-09-29 01:06:11 on dist-test-slave-7zdl"
server_key: "5203a836db46879e2be4946699289556"
server_key_iv: "8901e0fbbb5bc818a255ab19527f4288"
server_key_version: "encryptionkey@0"
I20260929 01:06:11.256327 19645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fb1f52d08d8e4a63bc7a702595b68d44"
format_stamp: "Formatted at 2026-09-29 01:06:11 on dist-test-slave-7zdl"
server_key: "5203a836db46879e2be4946699289556"
server_key_iv: "8901e0fbbb5bc818a255ab19527f4288"
server_key_version: "encryptionkey@0"
I20260929 01:06:11.259807 19645 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:06:11.262137 19660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:11.263475 19645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:06:11.263602 19645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "fb1f52d08d8e4a63bc7a702595b68d44"
format_stamp: "Formatted at 2026-09-29 01:06:11 on dist-test-slave-7zdl"
server_key: "5203a836db46879e2be4946699289556"
server_key_iv: "8901e0fbbb5bc818a255ab19527f4288"
server_key_version: "encryptionkey@0"
I20260929 01:06:11.263708 19645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:11.281170 19645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:11.284169 19645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:11.284359 19645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:11.284910 19645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:06:11.285681 19645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:06:11.285748 19645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:11.285818 19645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:06:11.285861 19645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:11.295053 19645 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38643
I20260929 01:06:11.295094 19773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38643 every 8 connection(s)
I20260929 01:06:11.295922 19645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:06:11.303500 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 19645
I20260929 01:06:11.303637 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:06:11.303968 392 external_mini_cluster.cc:1428] Setting key 7829821cf16cadb401cebe4cb302bf7c
Sep 29 01:06:11 dist-test-slave-7zdl krb5kdc[19538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:06:11.309113 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:11.297639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50817 (local address 127.0.98.62:34705)
0929 01:06:11.297782 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:11.297786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:11.298580 (+ 794us) server_negotiation.cc:407] Connection header received
0929 01:06:11.299474 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:11.299478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:11.299532 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:11.299618 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:11.300752 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.301293 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.302426 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.302663 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.302791 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:11.305303 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:11.305307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:11.305311 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:11.305337 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:11.306956 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:11.307503 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:11.307509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:11.307511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:11.307604 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:11.307906 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:11.307912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:11.307914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:11.308162 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:11.308266 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:11.308701 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 01:06:11.308854 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:06:11.309670 19776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:11.297984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34705 (local address 127.0.98.1:50817)
0929 01:06:11.298433 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:11.298479 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:11.299275 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:11.299788 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:11.299794 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:11.300146 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:11.300607 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:11.300619 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.301417 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.301421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:11.302303 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:11.302311 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.302852 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.302857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:11.303059 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:11.303605 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:11.303627 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:11.305164 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:11.307159 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:11.307164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:11.307175 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:11.307411 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:11.307694 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:11.307696 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:11.307697 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:11.307818 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:11.308260 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:11.308265 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:11.308482 (+ 217us) client_negotiation.cc:769] Sending connection context
0929 01:06:11.308624 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 01:06:11.308871 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":131,"threads_started":1}
I20260929 01:06:11.310850 19774 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34705
I20260929 01:06:11.311115 19774 heartbeater.cc:464] Registering TS with master...
I20260929 01:06:11.311671 19774 heartbeater.cc:511] Master 127.0.98.62:34705 requested a full tablet report, sending...
I20260929 01:06:11.313151 19583 ts_manager.cc:194] Registered new tserver with Master: fb1f52d08d8e4a63bc7a702595b68d44 (127.0.98.1:38643)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 01:06:11.314301 19583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50817
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 01:06:11.331905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34705
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:11.443701 19779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:11.443954 19779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:11.444000 19779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:11.447949 19779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:11.448004 19779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:11.448062 19779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:06:11.453287 19779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:11.454325 19779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:11.455293 19779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:11.465104 19784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:11.465095 19787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:11.465095 19785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:11.466421 19779 server_base.cc:1034] running on GCE node
I20260929 01:06:11.466951 19779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:11.467713 19779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:11.468887 19779 hybrid_clock.cc:648] HybridClock initialized: now 1790643971468871 us; error 32 us; skew 500 ppm
Sep 29 01:06:11 dist-test-slave-7zdl krb5kdc[19538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:11.471741 19779 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 01:06:11.472606 19779 webserver.cc:460] Webserver started at http://127.0.98.2:40413/ using document root <none> and password file <none>
I20260929 01:06:11.473204 19779 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:11.473294 19779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:11.473522 19779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:11.475610 19779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "2ec018dcd75a43d5b3b13d49528b2f28"
format_stamp: "Formatted at 2026-09-29 01:06:11 on dist-test-slave-7zdl"
server_key: "2f75af97e775a6877dee40f60e01ef90"
server_key_iv: "40f51d18bc47ec96a25496cc42d8e320"
server_key_version: "encryptionkey@0"
I20260929 01:06:11.476157 19779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2ec018dcd75a43d5b3b13d49528b2f28"
format_stamp: "Formatted at 2026-09-29 01:06:11 on dist-test-slave-7zdl"
server_key: "2f75af97e775a6877dee40f60e01ef90"
server_key_iv: "40f51d18bc47ec96a25496cc42d8e320"
server_key_version: "encryptionkey@0"
I20260929 01:06:11.479823 19779 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:06:11.482120 19794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:11.483331 19779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:06:11.483466 19779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "2ec018dcd75a43d5b3b13d49528b2f28"
format_stamp: "Formatted at 2026-09-29 01:06:11 on dist-test-slave-7zdl"
server_key: "2f75af97e775a6877dee40f60e01ef90"
server_key_iv: "40f51d18bc47ec96a25496cc42d8e320"
server_key_version: "encryptionkey@0"
I20260929 01:06:11.483572 19779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:11.522922 19779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:11.526266 19779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:11.526474 19779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:11.527101 19779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:06:11.527916 19779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:06:11.527987 19779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:11.528057 19779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:06:11.528101 19779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:11.537043 19779 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38765
I20260929 01:06:11.537091 19907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38765 every 8 connection(s)
I20260929 01:06:11.537951 19779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:06:11.538594 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 19779
I20260929 01:06:11.538753 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:06:11.539093 392 external_mini_cluster.cc:1428] Setting key 055f85bdcd5f8cad57c46adc242bc5ba
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 29 01:06:11 dist-test-slave-7zdl krb5kdc[19538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 01:06:11.554911 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:11.540088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57195 (local address 127.0.98.62:34705)
0929 01:06:11.540231 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:11.540234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:11.542298 (+ 2064us) server_negotiation.cc:407] Connection header received
0929 01:06:11.543472 (+ 1174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:11.543479 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:11.543546 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:11.543656 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:11.544976 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.545867 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.547176 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.547587 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.547708 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:11.550490 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:11.550494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:11.550496 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:11.550543 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:11.552305 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:11.553086 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:11.553089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:11.553090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:11.553167 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:11.553625 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:11.553644 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:11.553646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:11.553885 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:11.553998 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:11.554571 (+ 573us) server_negotiation.cc:299] Negotiation successful
0929 01:06:11.554703 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 01:06:11.555675 19910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:11.540544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34705 (local address 127.0.98.2:57195)
0929 01:06:11.542119 (+ 1575us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:11.542181 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:11.543247 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:11.543898 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:11.543905 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:11.544310 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:11.544818 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:11.544831 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.546068 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.546071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:11.547054 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:11.547063 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.547755 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.547758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:11.547916 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:11.548666 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:11.548684 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:11.550286 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:11.552455 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:11.552461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:11.552476 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:11.552901 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:11.553289 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:11.553294 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:11.553296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:11.553520 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:11.553995 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:11.554002 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:11.554312 (+ 310us) client_negotiation.cc:769] Sending connection context
0929 01:06:11.554519 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 01:06:11.554866 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1319,"thread_start_us":157,"threads_started":1}
I20260929 01:06:11.556792 19908 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34705
I20260929 01:06:11.557046 19908 heartbeater.cc:464] Registering TS with master...
I20260929 01:06:11.557538 19908 heartbeater.cc:511] Master 127.0.98.62:34705 requested a full tablet report, sending...
I20260929 01:06:11.558630 19583 ts_manager.cc:194] Registered new tserver with Master: 2ec018dcd75a43d5b3b13d49528b2f28 (127.0.98.2:38765)
I20260929 01:06:11.559515 19583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 01:06:11.569777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34705
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:11.678486 19913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:11.678719 19913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:11.678792 19913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:11.682551 19913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:11.682600 19913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:11.682720 19913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:06:11.687899 19913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:11.688874 19913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:11.689832 19913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:11.699019 19921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:11.699028 19919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:11.699040 19918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:11.699332 19913 server_base.cc:1034] running on GCE node
I20260929 01:06:11.699827 19913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:11.700487 19913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:11.701653 19913 hybrid_clock.cc:648] HybridClock initialized: now 1790643971701642 us; error 24 us; skew 500 ppm
Sep 29 01:06:11 dist-test-slave-7zdl krb5kdc[19538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:11.704831 19913 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 01:06:11.705921 19913 webserver.cc:460] Webserver started at http://127.0.98.3:37103/ using document root <none> and password file <none>
I20260929 01:06:11.706624 19913 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:11.706712 19913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:11.707015 19913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:11.709707 19913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "5c014ec9e13a4a558833d13e7a85d825"
format_stamp: "Formatted at 2026-09-29 01:06:11 on dist-test-slave-7zdl"
server_key: "604badd0011057bed78d322c93664458"
server_key_iv: "aa6a7a9ae74de8547a6699cf8ffeff9a"
server_key_version: "encryptionkey@0"
I20260929 01:06:11.710351 19913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5c014ec9e13a4a558833d13e7a85d825"
format_stamp: "Formatted at 2026-09-29 01:06:11 on dist-test-slave-7zdl"
server_key: "604badd0011057bed78d322c93664458"
server_key_iv: "aa6a7a9ae74de8547a6699cf8ffeff9a"
server_key_version: "encryptionkey@0"
I20260929 01:06:11.715008 19913 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 01:06:11.717262 19928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:11.718411 19913 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:06:11.718539 19913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "5c014ec9e13a4a558833d13e7a85d825"
format_stamp: "Formatted at 2026-09-29 01:06:11 on dist-test-slave-7zdl"
server_key: "604badd0011057bed78d322c93664458"
server_key_iv: "aa6a7a9ae74de8547a6699cf8ffeff9a"
server_key_version: "encryptionkey@0"
I20260929 01:06:11.718645 19913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:11.734290 19913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:11.737526 19913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:11.737732 19913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:11.738312 19913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:06:11.739162 19913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:06:11.739229 19913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:11.739303 19913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:06:11.739363 19913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:11.748764 19913 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38309
I20260929 01:06:11.748797 20041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38309 every 8 connection(s)
I20260929 01:06:11.749656 19913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:06:11.755688 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 19913
I20260929 01:06:11.755841 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:06:11.756172 392 external_mini_cluster.cc:1428] Setting key 4a6187fa2b3a7d94fda71806b94c6e72
Sep 29 01:06:11 dist-test-slave-7zdl krb5kdc[19538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:06:11.763522 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:11.751623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57761 (local address 127.0.98.62:34705)
0929 01:06:11.751802 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:11.751807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:11.752706 (+ 899us) server_negotiation.cc:407] Connection header received
0929 01:06:11.753664 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:11.753667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:11.753714 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:11.753794 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:11.755087 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.755765 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.757113 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.757390 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.757503 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:11.760006 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:11.760009 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:11.760012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:11.760044 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:11.761423 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:11.761964 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:11.761971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:11.761985 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:11.762067 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:11.762414 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:11.762417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:11.762419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:11.762601 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:11.762724 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:11.763243 (+ 519us) server_negotiation.cc:299] Negotiation successful
0929 01:06:11.763359 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 01:06:11.764290 20044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:11.752032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34705 (local address 127.0.98.3:57761)
0929 01:06:11.752532 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:11.752572 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:11.753477 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:11.754039 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:11.754046 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:11.754402 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:11.754946 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:11.754966 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.755914 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.755920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:11.756965 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:11.756994 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.757552 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.757557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:11.757730 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:11.758354 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:11.758380 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:11.759848 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:11.761589 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:11.761595 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:11.761608 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:11.761858 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:11.762163 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:11.762167 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:11.762170 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:11.762307 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:11.762721 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:11.762728 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:11.763015 (+ 287us) client_negotiation.cc:769] Sending connection context
0929 01:06:11.763203 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 01:06:11.763516 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":126,"threads_started":1}
I20260929 01:06:11.765311 20042 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34705
I20260929 01:06:11.765516 20042 heartbeater.cc:464] Registering TS with master...
I20260929 01:06:11.765926 20042 heartbeater.cc:511] Master 127.0.98.62:34705 requested a full tablet report, sending...
I20260929 01:06:11.766860 19583 ts_manager.cc:194] Registered new tserver with Master: 5c014ec9e13a4a558833d13e7a85d825 (127.0.98.3:38309)
I20260929 01:06:11.767472 19583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57761
I20260929 01:06:11.769829 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:06:11.784273 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:11.775423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59328 (local address 127.0.98.62:34705)
0929 01:06:11.775575 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:11.775581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:11.775719 (+ 138us) server_negotiation.cc:407] Connection header received
0929 01:06:11.775864 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:11.775870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:11.775927 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:11.776030 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:11.777045 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.777563 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.779073 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.779390 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.779492 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:11.780752 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:11.780758 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:11.780763 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:11.780793 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:11.782411 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:11.782840 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:11.782845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:11.782849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:11.782947 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:11.783230 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:11.783236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:11.783240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:11.783552 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:11.783698 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:11.783888 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 01:06:11.784039 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:06:11.787475 19583 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:06:11.792989 19583 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:06:11.810978 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:11.802844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38309 (local address 127.0.0.1:40932)
0929 01:06:11.803139 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:11.803181 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:11.803299 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:11.805636 (+ 2337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:11.805639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:11.805655 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:11.805748 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:11.805754 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.807098 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.807101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:11.808543 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:11.808553 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.810520 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.810524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:11.810607 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:11.810643 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 01:06:11.810708 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 01:06:11.810779 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":70,"thread_start_us":71,"threads_started":1}
I20260929 01:06:11.811244 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:11.804124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40336 (local address 127.0.98.2:38765)
0929 01:06:11.804574 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:11.804583 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:11.804602 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:06:11.804669 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:11.804672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:11.804827 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:11.804982 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:11.805304 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.806720 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.808851 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.809875 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.810008 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:11.810197 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:11.810885 (+ 688us) server_negotiation.cc:299] Negotiation successful
0929 01:06:11.811025 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":125,"threads_started":1}
I20260929 01:06:11.811640 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:11.803588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38765 (local address 127.0.0.1:40336)
0929 01:06:11.803816 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:11.803829 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:11.803930 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:11.804985 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:11.804990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:11.805012 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:11.805159 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:11.805168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.806952 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.806958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:11.808668 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:11.808682 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.810266 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.810272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:11.810429 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:11.810481 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 01:06:11.810599 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 01:06:11.810741 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":38,"thread_start_us":99,"threads_started":1}
I20260929 01:06:11.811924 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:11.802994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40932 (local address 127.0.98.3:38309)
0929 01:06:11.805282 (+ 2288us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:11.805292 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:11.805312 (+ 20us) server_negotiation.cc:407] Connection header received
0929 01:06:11.805377 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:11.805381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:11.805515 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:11.805665 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:11.805870 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.806954 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.808693 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.809519 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.809616 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:11.811555 (+ 1939us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:11.811601 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 01:06:11.811784 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2179,"thread_start_us":156,"threads_started":1}
I20260929 01:06:11.812083 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:11.804110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38132 (local address 127.0.98.1:38643)
0929 01:06:11.805940 (+ 1830us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:11.805955 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:11.805974 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:06:11.806042 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:11.806047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:11.806209 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:11.806346 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:11.807551 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.808651 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.810015 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.810823 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.810914 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:11.811049 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:11.811701 (+ 652us) server_negotiation.cc:299] Negotiation successful
0929 01:06:11.811872 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1728,"thread_start_us":113,"threads_started":1}
I20260929 01:06:11.813491 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:11.803921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38643 (local address 127.0.0.1:38132)
0929 01:06:11.804321 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:11.804337 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:11.804429 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:11.807215 (+ 2786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:11.807220 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:11.807240 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:11.807418 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:11.807425 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.808802 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.808805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:11.809890 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:11.809900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.811268 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.811275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:11.811475 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:11.811496 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 01:06:11.811557 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 01:06:11.811623 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":60,"thread_start_us":120,"threads_started":1}
I20260929 01:06:11.814759 19842 tablet_service.cc:1467] Processing CreateTablet for tablet 1b2ee2b3520e4de4a77a570b2002987d (DEFAULT_TABLE table=test-table [id=a444dd6e8c4c4a0aab454c8bf7b441a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:06:11.815274 19976 tablet_service.cc:1467] Processing CreateTablet for tablet 1b2ee2b3520e4de4a77a570b2002987d (DEFAULT_TABLE table=test-table [id=a444dd6e8c4c4a0aab454c8bf7b441a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:06:11.816252 19842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b2ee2b3520e4de4a77a570b2002987d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:11.816251 19976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b2ee2b3520e4de4a77a570b2002987d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:11.816474 19708 tablet_service.cc:1467] Processing CreateTablet for tablet 1b2ee2b3520e4de4a77a570b2002987d (DEFAULT_TABLE table=test-table [id=a444dd6e8c4c4a0aab454c8bf7b441a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:06:11.817535 19708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b2ee2b3520e4de4a77a570b2002987d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:11.824841 20060 tablet_bootstrap.cc:492] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825: Bootstrap starting.
I20260929 01:06:11.826678 20061 tablet_bootstrap.cc:492] T 1b2ee2b3520e4de4a77a570b2002987d P fb1f52d08d8e4a63bc7a702595b68d44: Bootstrap starting.
I20260929 01:06:11.826853 20060 tablet_bootstrap.cc:654] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:11.828100 20060 log.cc:824] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:11.828796 20059 tablet_bootstrap.cc:492] T 1b2ee2b3520e4de4a77a570b2002987d P 2ec018dcd75a43d5b3b13d49528b2f28: Bootstrap starting.
I20260929 01:06:11.828970 20061 tablet_bootstrap.cc:654] T 1b2ee2b3520e4de4a77a570b2002987d P fb1f52d08d8e4a63bc7a702595b68d44: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:11.830546 20060 tablet_bootstrap.cc:492] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825: No bootstrap required, opened a new log
I20260929 01:06:11.830839 20060 ts_tablet_manager.cc:1397] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260929 01:06:11.830904 20061 log.cc:824] T 1b2ee2b3520e4de4a77a570b2002987d P fb1f52d08d8e4a63bc7a702595b68d44: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:11.831509 20059 tablet_bootstrap.cc:654] T 1b2ee2b3520e4de4a77a570b2002987d P 2ec018dcd75a43d5b3b13d49528b2f28: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:11.832211 20059 log.cc:824] T 1b2ee2b3520e4de4a77a570b2002987d P 2ec018dcd75a43d5b3b13d49528b2f28: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:11.833962 20059 tablet_bootstrap.cc:492] T 1b2ee2b3520e4de4a77a570b2002987d P 2ec018dcd75a43d5b3b13d49528b2f28: No bootstrap required, opened a new log
I20260929 01:06:11.834167 20059 ts_tablet_manager.cc:1397] T 1b2ee2b3520e4de4a77a570b2002987d P 2ec018dcd75a43d5b3b13d49528b2f28: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 01:06:11.835778 20061 tablet_bootstrap.cc:492] T 1b2ee2b3520e4de4a77a570b2002987d P fb1f52d08d8e4a63bc7a702595b68d44: No bootstrap required, opened a new log
I20260929 01:06:11.835974 20061 ts_tablet_manager.cc:1397] T 1b2ee2b3520e4de4a77a570b2002987d P fb1f52d08d8e4a63bc7a702595b68d44: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260929 01:06:11.836261 20060 raft_consensus.cc:348] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c014ec9e13a4a558833d13e7a85d825" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38309 } } peers { permanent_uuid: "2ec018dcd75a43d5b3b13d49528b2f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38765 } } peers { permanent_uuid: "fb1f52d08d8e4a63bc7a702595b68d44" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38643 } }
I20260929 01:06:11.836567 20060 raft_consensus.cc:374] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:11.836632 20060 raft_consensus.cc:729] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c014ec9e13a4a558833d13e7a85d825, State: Initialized, Role: FOLLOWER
I20260929 01:06:11.837184 20060 consensus_queue.cc:260] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c014ec9e13a4a558833d13e7a85d825" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38309 } } peers { permanent_uuid: "2ec018dcd75a43d5b3b13d49528b2f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38765 } } peers { permanent_uuid: "fb1f52d08d8e4a63bc7a702595b68d44" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38643 } }
I20260929 01:06:11.838368 20060 ts_tablet_manager.cc:1428] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 01:06:11.838311 20059 raft_consensus.cc:348] T 1b2ee2b3520e4de4a77a570b2002987d P 2ec018dcd75a43d5b3b13d49528b2f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c014ec9e13a4a558833d13e7a85d825" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38309 } } peers { permanent_uuid: "2ec018dcd75a43d5b3b13d49528b2f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38765 } } peers { permanent_uuid: "fb1f52d08d8e4a63bc7a702595b68d44" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38643 } }
I20260929 01:06:11.838598 20059 raft_consensus.cc:374] T 1b2ee2b3520e4de4a77a570b2002987d P 2ec018dcd75a43d5b3b13d49528b2f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:11.838661 20059 raft_consensus.cc:729] T 1b2ee2b3520e4de4a77a570b2002987d P 2ec018dcd75a43d5b3b13d49528b2f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ec018dcd75a43d5b3b13d49528b2f28, State: Initialized, Role: FOLLOWER
I20260929 01:06:11.838735 20042 heartbeater.cc:503] Master 127.0.98.62:34705 was elected leader, sending a full tablet report...
I20260929 01:06:11.839154 20059 consensus_queue.cc:260] T 1b2ee2b3520e4de4a77a570b2002987d P 2ec018dcd75a43d5b3b13d49528b2f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c014ec9e13a4a558833d13e7a85d825" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38309 } } peers { permanent_uuid: "2ec018dcd75a43d5b3b13d49528b2f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38765 } } peers { permanent_uuid: "fb1f52d08d8e4a63bc7a702595b68d44" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38643 } }
I20260929 01:06:11.839920 19908 heartbeater.cc:503] Master 127.0.98.62:34705 was elected leader, sending a full tablet report...
I20260929 01:06:11.840183 20059 ts_tablet_manager.cc:1428] T 1b2ee2b3520e4de4a77a570b2002987d P 2ec018dcd75a43d5b3b13d49528b2f28: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 01:06:11.841318 20065 raft_consensus.cc:482] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:06:11.841478 20065 raft_consensus.cc:504] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c014ec9e13a4a558833d13e7a85d825" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38309 } } peers { permanent_uuid: "2ec018dcd75a43d5b3b13d49528b2f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38765 } } peers { permanent_uuid: "fb1f52d08d8e4a63bc7a702595b68d44" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38643 } }
I20260929 01:06:11.842636 20065 leader_election.cc:288] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ec018dcd75a43d5b3b13d49528b2f28 (127.0.98.2:38765), fb1f52d08d8e4a63bc7a702595b68d44 (127.0.98.1:38643)
I20260929 01:06:11.843768 20061 raft_consensus.cc:348] T 1b2ee2b3520e4de4a77a570b2002987d P fb1f52d08d8e4a63bc7a702595b68d44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c014ec9e13a4a558833d13e7a85d825" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38309 } } peers { permanent_uuid: "2ec018dcd75a43d5b3b13d49528b2f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38765 } } peers { permanent_uuid: "fb1f52d08d8e4a63bc7a702595b68d44" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38643 } }
I20260929 01:06:11.844087 20061 raft_consensus.cc:374] T 1b2ee2b3520e4de4a77a570b2002987d P fb1f52d08d8e4a63bc7a702595b68d44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:11.844154 20061 raft_consensus.cc:729] T 1b2ee2b3520e4de4a77a570b2002987d P fb1f52d08d8e4a63bc7a702595b68d44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1f52d08d8e4a63bc7a702595b68d44, State: Initialized, Role: FOLLOWER
I20260929 01:06:11.844583 20061 consensus_queue.cc:260] T 1b2ee2b3520e4de4a77a570b2002987d P fb1f52d08d8e4a63bc7a702595b68d44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c014ec9e13a4a558833d13e7a85d825" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38309 } } peers { permanent_uuid: "2ec018dcd75a43d5b3b13d49528b2f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38765 } } peers { permanent_uuid: "fb1f52d08d8e4a63bc7a702595b68d44" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38643 } }
I20260929 01:06:11.845235 19774 heartbeater.cc:503] Master 127.0.98.62:34705 was elected leader, sending a full tablet report...
I20260929 01:06:11.845553 20061 ts_tablet_manager.cc:1428] T 1b2ee2b3520e4de4a77a570b2002987d P fb1f52d08d8e4a63bc7a702595b68d44: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260929 01:06:11.848629 20044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:11.843026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38765 (local address 127.0.98.3:48457)
0929 01:06:11.843789 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:11.843808 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:11.843927 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:11.844564 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:11.844567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:11.844585 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:11.844730 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:11.844739 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.845894 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.845898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:11.847097 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:11.847106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.848136 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.848140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:11.848292 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:11.848323 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 01:06:11.848404 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 01:06:11.848482 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":1}
I20260929 01:06:11.849367 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:11.844096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48457 (local address 127.0.98.2:38765)
0929 01:06:11.844228 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:11.844234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:11.844249 (+ 15us) server_negotiation.cc:407] Connection header received
0929 01:06:11.844296 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:11.844299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:11.844349 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:11.844441 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:11.844893 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.845626 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.847236 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.847839 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.847979 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:11.848071 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:11.849151 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0929 01:06:11.849231 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 01:06:11.849882 19862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2ee2b3520e4de4a77a570b2002987d" candidate_uuid: "5c014ec9e13a4a558833d13e7a85d825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ec018dcd75a43d5b3b13d49528b2f28" is_pre_election: true
I20260929 01:06:11.850095 19862 raft_consensus.cc:2454] T 1b2ee2b3520e4de4a77a570b2002987d P 2ec018dcd75a43d5b3b13d49528b2f28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c014ec9e13a4a558833d13e7a85d825 in term 0.
I20260929 01:06:11.850538 19931 leader_election.cc:302] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ec018dcd75a43d5b3b13d49528b2f28, 5c014ec9e13a4a558833d13e7a85d825; no voters:
I20260929 01:06:11.850868 20065 raft_consensus.cc:2781] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:06:11.850956 20065 raft_consensus.cc:482] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:06:11.851008 20065 raft_consensus.cc:3037] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:11.851162 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:11.843418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32987 (local address 127.0.98.1:38643)
0929 01:06:11.843547 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:11.843552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:11.846008 (+ 2456us) server_negotiation.cc:407] Connection header received
0929 01:06:11.846227 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:11.846231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:11.846278 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:11.846334 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:11.846734 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.847508 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.849579 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.850370 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.850464 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:11.850546 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:11.850888 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 01:06:11.850999 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:06:11.851213 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:11.843318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38643 (local address 127.0.98.3:32987)
0929 01:06:11.845908 (+ 2590us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:11.845923 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:11.846101 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:11.846485 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:11.846488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:11.846505 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:11.846633 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:11.846639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.847653 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.847658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:11.849327 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:11.849341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.850597 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.850601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:11.850704 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:11.850730 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 01:06:11.850871 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 01:06:11.850973 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2526,"thread_start_us":106,"threads_started":1}
I20260929 01:06:11.851755 19728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2ee2b3520e4de4a77a570b2002987d" candidate_uuid: "5c014ec9e13a4a558833d13e7a85d825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1f52d08d8e4a63bc7a702595b68d44" is_pre_election: true
I20260929 01:06:11.852027 19728 raft_consensus.cc:2454] T 1b2ee2b3520e4de4a77a570b2002987d P fb1f52d08d8e4a63bc7a702595b68d44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c014ec9e13a4a558833d13e7a85d825 in term 0.
I20260929 01:06:11.852521 20065 raft_consensus.cc:504] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c014ec9e13a4a558833d13e7a85d825" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38309 } } peers { permanent_uuid: "2ec018dcd75a43d5b3b13d49528b2f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38765 } } peers { permanent_uuid: "fb1f52d08d8e4a63bc7a702595b68d44" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38643 } }
I20260929 01:06:11.852867 20065 leader_election.cc:288] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825 [CANDIDATE]: Term 1 election: Requested vote from peers 2ec018dcd75a43d5b3b13d49528b2f28 (127.0.98.2:38765), fb1f52d08d8e4a63bc7a702595b68d44 (127.0.98.1:38643)
I20260929 01:06:11.853255 19862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2ee2b3520e4de4a77a570b2002987d" candidate_uuid: "5c014ec9e13a4a558833d13e7a85d825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ec018dcd75a43d5b3b13d49528b2f28"
I20260929 01:06:11.853318 19728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2ee2b3520e4de4a77a570b2002987d" candidate_uuid: "5c014ec9e13a4a558833d13e7a85d825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1f52d08d8e4a63bc7a702595b68d44"
I20260929 01:06:11.853387 19862 raft_consensus.cc:3037] T 1b2ee2b3520e4de4a77a570b2002987d P 2ec018dcd75a43d5b3b13d49528b2f28 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:11.853453 19728 raft_consensus.cc:3037] T 1b2ee2b3520e4de4a77a570b2002987d P fb1f52d08d8e4a63bc7a702595b68d44 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:11.854779 19728 raft_consensus.cc:2454] T 1b2ee2b3520e4de4a77a570b2002987d P fb1f52d08d8e4a63bc7a702595b68d44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c014ec9e13a4a558833d13e7a85d825 in term 1.
I20260929 01:06:11.854779 19862 raft_consensus.cc:2454] T 1b2ee2b3520e4de4a77a570b2002987d P 2ec018dcd75a43d5b3b13d49528b2f28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c014ec9e13a4a558833d13e7a85d825 in term 1.
I20260929 01:06:11.855140 19930 leader_election.cc:302] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c014ec9e13a4a558833d13e7a85d825, fb1f52d08d8e4a63bc7a702595b68d44; no voters:
I20260929 01:06:11.855326 20065 raft_consensus.cc:2781] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:11.855561 20065 raft_consensus.cc:686] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825 [term 1 LEADER]: Becoming Leader. State: Replica: 5c014ec9e13a4a558833d13e7a85d825, State: Running, Role: LEADER
I20260929 01:06:11.855876 20065 consensus_queue.cc:237] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c014ec9e13a4a558833d13e7a85d825" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38309 } } peers { permanent_uuid: "2ec018dcd75a43d5b3b13d49528b2f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38765 } } peers { permanent_uuid: "fb1f52d08d8e4a63bc7a702595b68d44" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38643 } }
I20260929 01:06:11.859344 19583 catalog_manager.cc:5495] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c014ec9e13a4a558833d13e7a85d825 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5c014ec9e13a4a558833d13e7a85d825" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c014ec9e13a4a558833d13e7a85d825" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ec018dcd75a43d5b3b13d49528b2f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb1f52d08d8e4a63bc7a702595b68d44" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38643 } health_report { overall_health: UNKNOWN } } }
I20260929 01:06:11.881304 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:11.877681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40946 (local address 127.0.98.3:38309)
0929 01:06:11.877822 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:11.877826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:11.877842 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:06:11.877940 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:11.877942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:11.877999 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:11.878116 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:11.878370 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.878959 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.879816 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:11.880046 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:11.880158 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:11.880477 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:11.880917 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:11.881059 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:11.881094 (+ 35us) server_negotiation.cc:299] Negotiation successful
0929 01:06:11.881153 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 01:06:11.887595 19728 raft_consensus.cc:1261] T 1b2ee2b3520e4de4a77a570b2002987d P fb1f52d08d8e4a63bc7a702595b68d44 [term 1 FOLLOWER]: Refusing update from remote peer 5c014ec9e13a4a558833d13e7a85d825: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:06:11.887583 19862 raft_consensus.cc:1261] T 1b2ee2b3520e4de4a77a570b2002987d P 2ec018dcd75a43d5b3b13d49528b2f28 [term 1 FOLLOWER]: Refusing update from remote peer 5c014ec9e13a4a558833d13e7a85d825: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:06:11.888377 20065 consensus_queue.cc:1035] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ec018dcd75a43d5b3b13d49528b2f28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:06:11.888571 20069 consensus_queue.cc:1035] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1f52d08d8e4a63bc7a702595b68d44" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:06:11.896417 20072 mvcc.cc:205] Tried to move back new op lower bound from 7334477708843270144 to 7334477708724383744. Current Snapshot: MvccSnapshot[applied={T|T < 7334477708843270144}]
I20260929 01:06:11.896829 20074 mvcc.cc:205] Tried to move back new op lower bound from 7334477708843270144 to 7334477708724383744. Current Snapshot: MvccSnapshot[applied={T|T < 7334477708843270144}]
I20260929 01:06:11.907222 19583 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:06:11.907351 20073 mvcc.cc:205] Tried to move back new op lower bound from 7334477708843270144 to 7334477708724383744. Current Snapshot: MvccSnapshot[applied={T|T < 7334477708843270144}]
I20260929 01:06:11.907421 19583 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:06:11.910068 19583 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 589c66e4515e4fc0bc396d103d30d8db: Sending DeleteTablet for 3 replicas of tablet 1b2ee2b3520e4de4a77a570b2002987d
I20260929 01:06:11.911094 19976 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2ee2b3520e4de4a77a570b2002987d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:06:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40932
I20260929 01:06:11.911142 19708 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2ee2b3520e4de4a77a570b2002987d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:06:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38132
I20260929 01:06:11.911623 20081 tablet_replica.cc:330] T 1b2ee2b3520e4de4a77a570b2002987d P fb1f52d08d8e4a63bc7a702595b68d44: stopping tablet replica
I20260929 01:06:11.911641 19842 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2ee2b3520e4de4a77a570b2002987d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:06:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40336
I20260929 01:06:11.911942 20081 raft_consensus.cc:2229] T 1b2ee2b3520e4de4a77a570b2002987d P fb1f52d08d8e4a63bc7a702595b68d44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:06:11.912191 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 19645
I20260929 01:06:11.912262 20081 raft_consensus.cc:2258] T 1b2ee2b3520e4de4a77a570b2002987d P fb1f52d08d8e4a63bc7a702595b68d44 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 01:06:11.918985 19568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38643 (error 108)
I20260929 01:06:11.919123 20083 tablet_replica.cc:330] T 1b2ee2b3520e4de4a77a570b2002987d P 2ec018dcd75a43d5b3b13d49528b2f28: stopping tablet replica
I20260929 01:06:11.919466 20083 raft_consensus.cc:2229] T 1b2ee2b3520e4de4a77a570b2002987d P 2ec018dcd75a43d5b3b13d49528b2f28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:06:11.919791 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 19779
I20260929 01:06:11.919826 20083 raft_consensus.cc:2258] T 1b2ee2b3520e4de4a77a570b2002987d P 2ec018dcd75a43d5b3b13d49528b2f28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:06:11.920208 20082 tablet_replica.cc:330] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825: stopping tablet replica
I20260929 01:06:11.920526 20082 raft_consensus.cc:2229] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825 [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:06:11.920903 20082 raft_consensus.cc:2258] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:06:11.923502 20082 ts_tablet_manager.cc:1905] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:06:11.926245 20082 ts_tablet_manager.cc:1918] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:06:11.926338 20082 log.cc:1196] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/1b2ee2b3520e4de4a77a570b2002987d
I20260929 01:06:11.926661 20082 ts_tablet_manager.cc:1939] T 1b2ee2b3520e4de4a77a570b2002987d P 5c014ec9e13a4a558833d13e7a85d825: Deleting consensus metadata
I20260929 01:06:11.926883 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 19913
I20260929 01:06:11.929104 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:11.928792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38643 (local address 127.0.0.1:38146)
0929 01:06:11.928928 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:11.929002 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 01:06:11.929104 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:11.928855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38765 (local address 127.0.0.1:40348)
0929 01:06:11.929003 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:11.929040 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":3}
W20260929 01:06:11.929292 19569 catalog_manager.cc:4584] TS 2ec018dcd75a43d5b3b13d49528b2f28 (127.0.98.2:38765): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b2ee2b3520e4de4a77a570b2002987d: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38765: connect: Connection refused (error 111)
I20260929 01:06:11.932619 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 19551
2026-09-29T01:06:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2254 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T01:06:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:11Z Disabled control of system clock
I20260929 01:06:11.962270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46511
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 01:06:12.071161 20088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:12.071417 20088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:12.071451 20088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:12.075440 20088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:12.075500 20088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:12.075515 20088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:12.075533 20088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:12.075548 20088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:12.080369 20088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46511
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:12.081398 20088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:12.082731 20088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:12.091497 20096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:12.091483 20094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:12.091473 20088 server_base.cc:1034] running on GCE node
W20260929 01:06:12.091464 20093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:12.092621 20088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:12.093732 20088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:12.094892 20088 hybrid_clock.cc:648] HybridClock initialized: now 1790643972094874 us; error 35 us; skew 500 ppm
I20260929 01:06:12.096771 20088 webserver.cc:460] Webserver started at http://127.0.98.62:33591/ using document root <none> and password file <none>
I20260929 01:06:12.097361 20088 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:12.097455 20088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:12.097669 20088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:12.099792 20088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "4d384c50beb44507a82ca6ecf95c8011"
format_stamp: "Formatted at 2026-09-29 01:06:12 on dist-test-slave-7zdl"
server_key: "dbf6b7de7ce4df1073a373c1daad6c64"
server_key_iv: "27f8e478d00ffd6550ce84a0dd49a67e"
server_key_version: "encryptionkey@0"
I20260929 01:06:12.100358 20088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "4d384c50beb44507a82ca6ecf95c8011"
format_stamp: "Formatted at 2026-09-29 01:06:12 on dist-test-slave-7zdl"
server_key: "dbf6b7de7ce4df1073a373c1daad6c64"
server_key_iv: "27f8e478d00ffd6550ce84a0dd49a67e"
server_key_version: "encryptionkey@0"
I20260929 01:06:12.104074 20088 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:06:12.106460 20102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:12.107688 20088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:06:12.107828 20088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "4d384c50beb44507a82ca6ecf95c8011"
format_stamp: "Formatted at 2026-09-29 01:06:12 on dist-test-slave-7zdl"
server_key: "dbf6b7de7ce4df1073a373c1daad6c64"
server_key_iv: "27f8e478d00ffd6550ce84a0dd49a67e"
server_key_version: "encryptionkey@0"
I20260929 01:06:12.107939 20088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:12.125344 20088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:12.126060 20088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:12.126258 20088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:12.133123 20088 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46511
I20260929 01:06:12.133147 20154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46511 every 8 connection(s)
I20260929 01:06:12.134061 20088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:12.137089 20155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:12.137748 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 20088
I20260929 01:06:12.137883 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:12.138203 392 external_mini_cluster.cc:1428] Setting key f1dc9df456cef53a598959ebf087464e
I20260929 01:06:12.143494 20155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d384c50beb44507a82ca6ecf95c8011: Bootstrap starting.
I20260929 01:06:12.146124 20155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d384c50beb44507a82ca6ecf95c8011: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:12.147001 20155 log.cc:824] T 00000000000000000000000000000000 P 4d384c50beb44507a82ca6ecf95c8011: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:12.147518 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:12.139642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37368 (local address 127.0.98.62:46511)
0929 01:06:12.140261 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:12.140276 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:12.140312 (+ 36us) server_negotiation.cc:407] Connection header received
0929 01:06:12.140951 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:12.140967 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:12.141355 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:12.141629 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:12.141949 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.142987 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.144109 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.144564 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.144676 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:12.145589 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:12.145594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:12.145605 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:12.146021 (+ 416us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:12.146076 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:12.146095 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:12.146171 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:12.146371 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 01:06:12.146655 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":128,"threads_started":1}
I20260929 01:06:12.149037 20155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d384c50beb44507a82ca6ecf95c8011: No bootstrap required, opened a new log
I20260929 01:06:12.152441 20155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d384c50beb44507a82ca6ecf95c8011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d384c50beb44507a82ca6ecf95c8011" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46511 } }
I20260929 01:06:12.152676 20155 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d384c50beb44507a82ca6ecf95c8011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:12.152760 20155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d384c50beb44507a82ca6ecf95c8011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d384c50beb44507a82ca6ecf95c8011, State: Initialized, Role: FOLLOWER
I20260929 01:06:12.153179 20155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d384c50beb44507a82ca6ecf95c8011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d384c50beb44507a82ca6ecf95c8011" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46511 } }
I20260929 01:06:12.153329 20155 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d384c50beb44507a82ca6ecf95c8011 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:12.153419 20155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d384c50beb44507a82ca6ecf95c8011 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:12.153542 20155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d384c50beb44507a82ca6ecf95c8011 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:12.154430 20155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d384c50beb44507a82ca6ecf95c8011 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d384c50beb44507a82ca6ecf95c8011" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46511 } }
I20260929 01:06:12.154762 20155 leader_election.cc:302] T 00000000000000000000000000000000 P 4d384c50beb44507a82ca6ecf95c8011 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d384c50beb44507a82ca6ecf95c8011; no voters:
I20260929 01:06:12.155200 20155 leader_election.cc:288] T 00000000000000000000000000000000 P 4d384c50beb44507a82ca6ecf95c8011 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:12.155304 20160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d384c50beb44507a82ca6ecf95c8011 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:12.155496 20160 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d384c50beb44507a82ca6ecf95c8011 [term 1 LEADER]: Becoming Leader. State: Replica: 4d384c50beb44507a82ca6ecf95c8011, State: Running, Role: LEADER
I20260929 01:06:12.155797 20160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d384c50beb44507a82ca6ecf95c8011 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d384c50beb44507a82ca6ecf95c8011" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46511 } }
I20260929 01:06:12.156361 20155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d384c50beb44507a82ca6ecf95c8011 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:12.157380 20161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d384c50beb44507a82ca6ecf95c8011 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d384c50beb44507a82ca6ecf95c8011" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d384c50beb44507a82ca6ecf95c8011" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46511 } } }
I20260929 01:06:12.157506 20161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d384c50beb44507a82ca6ecf95c8011 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:12.157816 20162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d384c50beb44507a82ca6ecf95c8011 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d384c50beb44507a82ca6ecf95c8011. Latest consensus state: current_term: 1 leader_uuid: "4d384c50beb44507a82ca6ecf95c8011" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d384c50beb44507a82ca6ecf95c8011" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46511 } } }
I20260929 01:06:12.157902 20162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d384c50beb44507a82ca6ecf95c8011 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:12.158011 20169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:12.160895 20169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:12.166371 20169 catalog_manager.cc:1347] Generated new cluster ID: 39df361d4f464682852a022919f051bf
I20260929 01:06:12.166440 20169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:12.191191 20169 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:12.192154 20169 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:12.199496 20169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d384c50beb44507a82ca6ecf95c8011: Generated new TSK 0
I20260929 01:06:12.200163 20169 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:12.208689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46511
--builtin_ntp_servers=127.0.98.20:41541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 01:06:12.328013 20179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:12.328289 20179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:12.328361 20179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:12.332419 20179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:12.332465 20179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:12.332568 20179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 01:06:12.337363 20179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:12.338414 20179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:12.339864 20179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:12.349293 20187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:12.349305 20184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:12.349351 20185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:12.349629 20179 server_base.cc:1034] running on GCE node
I20260929 01:06:12.350087 20179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:12.350685 20179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:12.351840 20179 hybrid_clock.cc:648] HybridClock initialized: now 1790643972351818 us; error 32 us; skew 500 ppm
I20260929 01:06:12.353675 20179 webserver.cc:460] Webserver started at http://127.0.98.1:34797/ using document root <none> and password file <none>
I20260929 01:06:12.354229 20179 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:12.354288 20179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:12.354539 20179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:12.356585 20179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-0/data/instance:
uuid: "3f521b440d444258979d80243563fecf"
format_stamp: "Formatted at 2026-09-29 01:06:12 on dist-test-slave-7zdl"
server_key: "8cf4ffde9b0e1ad18db4d3c30519a7d6"
server_key_iv: "9dc252379e70af41eb41b6f9ff1462ce"
server_key_version: "encryptionkey@0"
I20260929 01:06:12.357125 20179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3f521b440d444258979d80243563fecf"
format_stamp: "Formatted at 2026-09-29 01:06:12 on dist-test-slave-7zdl"
server_key: "8cf4ffde9b0e1ad18db4d3c30519a7d6"
server_key_iv: "9dc252379e70af41eb41b6f9ff1462ce"
server_key_version: "encryptionkey@0"
I20260929 01:06:12.360761 20179 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:06:12.363056 20193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:12.364241 20179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:06:12.364385 20179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-0/wal
uuid: "3f521b440d444258979d80243563fecf"
format_stamp: "Formatted at 2026-09-29 01:06:12 on dist-test-slave-7zdl"
server_key: "8cf4ffde9b0e1ad18db4d3c30519a7d6"
server_key_iv: "9dc252379e70af41eb41b6f9ff1462ce"
server_key_version: "encryptionkey@0"
I20260929 01:06:12.364497 20179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:12.384258 20179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:12.384925 20179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:12.385118 20179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:12.385807 20179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:06:12.386726 20179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:06:12.386796 20179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:12.386879 20179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:06:12.386931 20179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:12.395985 20179 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33531
I20260929 01:06:12.396020 20306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33531 every 8 connection(s)
I20260929 01:06:12.396888 20179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-0/data/info.pb
I20260929 01:06:12.405529 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 20179
I20260929 01:06:12.405681 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-0/wal/instance
I20260929 01:06:12.405962 392 external_mini_cluster.cc:1428] Setting key a6ded5f4b12430fba79ef9e92f338dfc
I20260929 01:06:12.406083 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:12.398657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57279 (local address 127.0.98.62:46511)
0929 01:06:12.398831 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:12.398835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:12.399886 (+ 1051us) server_negotiation.cc:407] Connection header received
0929 01:06:12.401291 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:12.401293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:12.401342 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:12.401414 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:12.402353 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.402921 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.404075 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.404298 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.404375 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:12.405560 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:12.405563 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:12.405565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:12.405578 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:12.405599 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:12.405606 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:12.405684 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:12.405865 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 01:06:12.405934 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 01:06:12.407245 20309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:12.399139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46511 (local address 127.0.98.1:57279)
0929 01:06:12.399758 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:12.399799 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:12.401103 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:12.401579 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:12.401585 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:12.401677 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:12.402224 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:12.402238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.403070 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.403072 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:12.403963 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:12.403972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.404430 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.404432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:12.404609 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:12.405238 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:06:12.405239 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:12.405282 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:06:12.405349 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:12.405698 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:12.405711 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:12.405720 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 01:06:12.405864 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 01:06:12.406168 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":104,"threads_started":1}
I20260929 01:06:12.408135 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46511
--builtin_ntp_servers=127.0.98.20:41541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 01:06:12.408483 20307 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46511
I20260929 01:06:12.408730 20307 heartbeater.cc:464] Registering TS with master...
I20260929 01:06:12.409241 20307 heartbeater.cc:511] Master 127.0.98.62:46511 requested a full tablet report, sending...
I20260929 01:06:12.411055 20119 ts_manager.cc:194] Registered new tserver with Master: 3f521b440d444258979d80243563fecf (127.0.98.1:33531)
I20260929 01:06:12.412233 20119 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:57279
W20260929 01:06:12.517688 20310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:12.517957 20310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:12.518028 20310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:12.522511 20310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:12.522600 20310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:12.522711 20310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 01:06:12.527405 20310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:12.528463 20310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:12.529846 20310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:12.539415 20318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:12.539482 20315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:12.539590 20310 server_base.cc:1034] running on GCE node
W20260929 01:06:12.539696 20316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:12.540122 20310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:12.540783 20310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:12.542091 20310 hybrid_clock.cc:648] HybridClock initialized: now 1790643972542073 us; error 38 us; skew 500 ppm
I20260929 01:06:12.544162 20310 webserver.cc:460] Webserver started at http://127.0.98.2:44595/ using document root <none> and password file <none>
I20260929 01:06:12.544796 20310 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:12.544899 20310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:12.545150 20310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:12.547420 20310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-1/data/instance:
uuid: "d49042bd14de4c10a2497b53319da7a3"
format_stamp: "Formatted at 2026-09-29 01:06:12 on dist-test-slave-7zdl"
server_key: "2feb746a911e58b5c1ab54a9ebd19008"
server_key_iv: "ba103f586ed213651f10a032194983dd"
server_key_version: "encryptionkey@0"
I20260929 01:06:12.548043 20310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d49042bd14de4c10a2497b53319da7a3"
format_stamp: "Formatted at 2026-09-29 01:06:12 on dist-test-slave-7zdl"
server_key: "2feb746a911e58b5c1ab54a9ebd19008"
server_key_iv: "ba103f586ed213651f10a032194983dd"
server_key_version: "encryptionkey@0"
I20260929 01:06:12.551959 20310 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:06:12.554534 20324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:12.555852 20310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:06:12.555997 20310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-1/wal
uuid: "d49042bd14de4c10a2497b53319da7a3"
format_stamp: "Formatted at 2026-09-29 01:06:12 on dist-test-slave-7zdl"
server_key: "2feb746a911e58b5c1ab54a9ebd19008"
server_key_iv: "ba103f586ed213651f10a032194983dd"
server_key_version: "encryptionkey@0"
I20260929 01:06:12.556116 20310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:12.582590 20310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:12.583401 20310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:12.583622 20310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:12.584241 20310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:06:12.585218 20310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:06:12.585295 20310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:12.585387 20310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:06:12.585472 20310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:12.594669 20310 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34115
I20260929 01:06:12.594698 20437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34115 every 8 connection(s)
I20260929 01:06:12.595621 20310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-1/data/info.pb
I20260929 01:06:12.604614 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:12.597483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35639 (local address 127.0.98.62:46511)
0929 01:06:12.597656 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:12.597661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:12.598756 (+ 1095us) server_negotiation.cc:407] Connection header received
0929 01:06:12.600124 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:12.600127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:12.600175 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:12.600260 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:12.601057 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.601600 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.602702 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.603051 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.603143 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:12.604191 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:12.604194 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:12.604196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:12.604208 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:12.604226 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:12.604234 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:12.604295 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:12.604438 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 01:06:12.604493 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 01:06:12.605388 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:12.597933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46511 (local address 127.0.98.2:35639)
0929 01:06:12.598581 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:12.598618 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:12.599917 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:12.600407 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:12.600413 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:12.600490 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:12.600939 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:12.600950 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.601721 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.601723 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:12.602572 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:12.602580 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.603190 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.603193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:12.603328 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:12.603951 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:06:12.603952 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:12.603992 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:06:12.604052 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:12.604308 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:12.604315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:12.604319 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 01:06:12.604410 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 01:06:12.604613 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":112,"threads_started":1}
I20260929 01:06:12.605773 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 20310
I20260929 01:06:12.605909 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-1/wal/instance
I20260929 01:06:12.606261 392 external_mini_cluster.cc:1428] Setting key 05c15e40bb34729feb817e83c1fbba22
I20260929 01:06:12.606771 20438 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46511
I20260929 01:06:12.607066 20438 heartbeater.cc:464] Registering TS with master...
I20260929 01:06:12.607661 20438 heartbeater.cc:511] Master 127.0.98.62:46511 requested a full tablet report, sending...
I20260929 01:06:12.608408 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46511
--builtin_ntp_servers=127.0.98.20:41541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 01:06:12.608704 20119 ts_manager.cc:194] Registered new tserver with Master: d49042bd14de4c10a2497b53319da7a3 (127.0.98.2:34115)
I20260929 01:06:12.609256 20119 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:35639
W20260929 01:06:12.716270 20441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:12.716502 20441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:12.716537 20441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:12.720548 20441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:12.720594 20441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:12.720649 20441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 01:06:12.725239 20441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:12.726249 20441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:12.727582 20441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:12.736845 20449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:12.736891 20446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:12.736900 20447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:12.737319 20441 server_base.cc:1034] running on GCE node
I20260929 01:06:12.737675 20441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:12.738250 20441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:12.739454 20441 hybrid_clock.cc:648] HybridClock initialized: now 1790643972739424 us; error 46 us; skew 500 ppm
I20260929 01:06:12.741331 20441 webserver.cc:460] Webserver started at http://127.0.98.3:43105/ using document root <none> and password file <none>
I20260929 01:06:12.741915 20441 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:12.742022 20441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:12.742269 20441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:12.744494 20441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-2/data/instance:
uuid: "b0c4ca74cab04690a326001e8d77ae5d"
format_stamp: "Formatted at 2026-09-29 01:06:12 on dist-test-slave-7zdl"
server_key: "962fed77f1721b1d5bbf4c385ea58ede"
server_key_iv: "c7e8b2ffc4584f4748a8934fa04551d8"
server_key_version: "encryptionkey@0"
I20260929 01:06:12.745095 20441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b0c4ca74cab04690a326001e8d77ae5d"
format_stamp: "Formatted at 2026-09-29 01:06:12 on dist-test-slave-7zdl"
server_key: "962fed77f1721b1d5bbf4c385ea58ede"
server_key_iv: "c7e8b2ffc4584f4748a8934fa04551d8"
server_key_version: "encryptionkey@0"
I20260929 01:06:12.749037 20441 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:06:12.751574 20455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:12.752813 20441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:06:12.752943 20441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-2/wal
uuid: "b0c4ca74cab04690a326001e8d77ae5d"
format_stamp: "Formatted at 2026-09-29 01:06:12 on dist-test-slave-7zdl"
server_key: "962fed77f1721b1d5bbf4c385ea58ede"
server_key_iv: "c7e8b2ffc4584f4748a8934fa04551d8"
server_key_version: "encryptionkey@0"
I20260929 01:06:12.753100 20441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:12.785313 20441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:12.786005 20441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:12.786243 20441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:12.786963 20441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 01:06:12.787887 20441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:06:12.787982 20441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:12.788049 20441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 01:06:12.788108 20441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:12.796988 20441 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41025
I20260929 01:06:12.797006 20568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41025 every 8 connection(s)
I20260929 01:06:12.797869 20441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-2/data/info.pb
I20260929 01:06:12.804889 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 20441
I20260929 01:06:12.805016 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-2/wal/instance
I20260929 01:06:12.805339 392 external_mini_cluster.cc:1428] Setting key bc05c75ddb58313771956612748fa4f4
I20260929 01:06:12.807058 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:12.799581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42801 (local address 127.0.98.62:46511)
0929 01:06:12.799726 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:12.799729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:12.800677 (+ 948us) server_negotiation.cc:407] Connection header received
0929 01:06:12.802028 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:12.802032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:12.802081 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:12.802155 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:12.803089 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.803615 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.804677 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.804887 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.805101 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:12.806389 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:12.806393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:12.806396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:12.806413 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:12.806440 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:12.806455 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:12.806567 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:12.806746 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 01:06:12.806841 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 01:06:12.807806 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:12.799894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46511 (local address 127.0.98.3:42801)
0929 01:06:12.800554 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:12.800592 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:12.801809 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:12.802316 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:12.802322 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:12.802402 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:12.802943 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:12.802958 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.803715 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.803718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:12.804574 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:12.804582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.805174 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.805178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:12.805377 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:12.806102 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 01:06:12.806104 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:12.806156 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 01:06:12.806215 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:12.806569 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:12.806581 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:12.806590 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 01:06:12.806743 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 01:06:12.807044 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":109,"threads_started":1}
I20260929 01:06:12.808745 20569 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46511
I20260929 01:06:12.808984 20569 heartbeater.cc:464] Registering TS with master...
I20260929 01:06:12.809479 20569 heartbeater.cc:511] Master 127.0.98.62:46511 requested a full tablet report, sending...
I20260929 01:06:12.810472 20119 ts_manager.cc:194] Registered new tserver with Master: b0c4ca74cab04690a326001e8d77ae5d (127.0.98.3:41025)
I20260929 01:06:12.811223 20119 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:42801
I20260929 01:06:12.819226 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 01:06:12.824728 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:12.821071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37376 (local address 127.0.98.62:46511)
0929 01:06:12.821264 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:12.821270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:12.821507 (+ 237us) server_negotiation.cc:407] Connection header received
0929 01:06:12.821594 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:12.821597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:12.821640 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:12.821708 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:12.822014 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.822516 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.823358 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.823562 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.823636 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:12.824342 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:12.824345 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:12.824347 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:12.824357 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:12.824373 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:12.824381 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:12.824436 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:12.824505 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 01:06:12.824549 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 01:06:12.827423 20119 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37376:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 01:06:12.830332 20119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 01:06:12.844588 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:12.838584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34115 (local address 127.0.0.1:53150)
0929 01:06:12.838901 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:12.838943 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:12.839058 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:12.840117 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:12.840124 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:12.840144 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:12.840302 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:12.840309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.841769 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.841772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:12.843102 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:12.843112 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.844143 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.844150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:12.844261 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:12.844286 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 01:06:12.844334 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 01:06:12.844434 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":86,"thread_start_us":72,"threads_started":1}
I20260929 01:06:12.844599 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:12.839003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49344 (local address 127.0.98.3:41025)
0929 01:06:12.839483 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:12.839497 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:12.839886 (+ 389us) server_negotiation.cc:407] Connection header received
0929 01:06:12.839957 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:12.839964 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:12.840145 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:12.840285 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:12.840665 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.841538 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.843399 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.843916 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.844052 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:12.844238 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:12.844334 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 01:06:12.844434 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":152,"threads_started":1}
I20260929 01:06:12.844890 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:12.839284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53150 (local address 127.0.98.2:34115)
0929 01:06:12.839599 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:12.839612 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:12.839647 (+ 35us) server_negotiation.cc:407] Connection header received
0929 01:06:12.839732 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:12.839740 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:12.839952 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:12.840445 (+ 493us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:12.840585 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.841598 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.843229 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.843991 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.844425 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:12.844589 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:12.844634 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 01:06:12.844734 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":135,"threads_started":1}
I20260929 01:06:12.844898 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:12.839321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33531 (local address 127.0.0.1:50138)
0929 01:06:12.840157 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:12.840172 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:12.840256 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:12.840853 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:12.840856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:12.840872 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:12.840958 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:12.840964 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.842167 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.842174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:12.843304 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:12.843314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.844527 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.844531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:12.844613 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:12.844644 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 01:06:12.844714 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 01:06:12.844774 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":41,"thread_start_us":92,"threads_started":1}
I20260929 01:06:12.845305 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:12.839708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50138 (local address 127.0.98.1:33531)
0929 01:06:12.840296 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:12.840312 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:12.840334 (+ 22us) server_negotiation.cc:407] Connection header received
0929 01:06:12.840406 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:12.840410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:12.840547 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:12.840710 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:12.841087 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.842058 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.843439 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.844188 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.844277 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:12.844457 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:12.845057 (+ 600us) server_negotiation.cc:299] Negotiation successful
0929 01:06:12.845158 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":114,"threads_started":1}
I20260929 01:06:12.845614 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:12.838858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41025 (local address 127.0.0.1:49344)
0929 01:06:12.839697 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:12.839711 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:12.839799 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:12.840426 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:12.840429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:12.840445 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:12.840528 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:12.840533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.841647 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.841654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:12.843294 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:12.843300 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.844051 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.844054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:12.844154 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:12.844189 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 01:06:12.844304 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 01:06:12.845468 (+ 1164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":72,"thread_start_us":94,"threads_started":1}
I20260929 01:06:12.847921 20503 tablet_service.cc:1467] Processing CreateTablet for tablet 3bcf20135d654c72ac0ef54d4cb4ef19 (DEFAULT_TABLE table=test-table [id=d44e9e6f4b934e03ba2a68fd63618b9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:06:12.847868 20372 tablet_service.cc:1467] Processing CreateTablet for tablet 3bcf20135d654c72ac0ef54d4cb4ef19 (DEFAULT_TABLE table=test-table [id=d44e9e6f4b934e03ba2a68fd63618b9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:06:12.847868 20241 tablet_service.cc:1467] Processing CreateTablet for tablet 3bcf20135d654c72ac0ef54d4cb4ef19 (DEFAULT_TABLE table=test-table [id=d44e9e6f4b934e03ba2a68fd63618b9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 01:06:12.848827 20503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bcf20135d654c72ac0ef54d4cb4ef19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:12.848851 20372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bcf20135d654c72ac0ef54d4cb4ef19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:12.848851 20241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bcf20135d654c72ac0ef54d4cb4ef19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:12.855057 20586 tablet_bootstrap.cc:492] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d: Bootstrap starting.
I20260929 01:06:12.856102 20587 tablet_bootstrap.cc:492] T 3bcf20135d654c72ac0ef54d4cb4ef19 P 3f521b440d444258979d80243563fecf: Bootstrap starting.
I20260929 01:06:12.856367 20588 tablet_bootstrap.cc:492] T 3bcf20135d654c72ac0ef54d4cb4ef19 P d49042bd14de4c10a2497b53319da7a3: Bootstrap starting.
I20260929 01:06:12.857443 20586 tablet_bootstrap.cc:654] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:12.858204 20587 tablet_bootstrap.cc:654] T 3bcf20135d654c72ac0ef54d4cb4ef19 P 3f521b440d444258979d80243563fecf: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:12.858291 20586 log.cc:824] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:12.858317 20588 tablet_bootstrap.cc:654] T 3bcf20135d654c72ac0ef54d4cb4ef19 P d49042bd14de4c10a2497b53319da7a3: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:12.859058 20587 log.cc:824] T 3bcf20135d654c72ac0ef54d4cb4ef19 P 3f521b440d444258979d80243563fecf: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:12.859206 20588 log.cc:824] T 3bcf20135d654c72ac0ef54d4cb4ef19 P d49042bd14de4c10a2497b53319da7a3: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:12.860492 20586 tablet_bootstrap.cc:492] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d: No bootstrap required, opened a new log
I20260929 01:06:12.860674 20586 ts_tablet_manager.cc:1397] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 01:06:12.861578 20587 tablet_bootstrap.cc:492] T 3bcf20135d654c72ac0ef54d4cb4ef19 P 3f521b440d444258979d80243563fecf: No bootstrap required, opened a new log
I20260929 01:06:12.861753 20587 ts_tablet_manager.cc:1397] T 3bcf20135d654c72ac0ef54d4cb4ef19 P 3f521b440d444258979d80243563fecf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 01:06:12.863870 20588 tablet_bootstrap.cc:492] T 3bcf20135d654c72ac0ef54d4cb4ef19 P d49042bd14de4c10a2497b53319da7a3: No bootstrap required, opened a new log
I20260929 01:06:12.864058 20588 ts_tablet_manager.cc:1397] T 3bcf20135d654c72ac0ef54d4cb4ef19 P d49042bd14de4c10a2497b53319da7a3: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260929 01:06:12.864945 20586 raft_consensus.cc:348] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49042bd14de4c10a2497b53319da7a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } } peers { permanent_uuid: "3f521b440d444258979d80243563fecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33531 } } peers { permanent_uuid: "b0c4ca74cab04690a326001e8d77ae5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41025 } }
I20260929 01:06:12.865216 20586 raft_consensus.cc:374] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:12.865270 20586 raft_consensus.cc:729] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c4ca74cab04690a326001e8d77ae5d, State: Initialized, Role: FOLLOWER
I20260929 01:06:12.865427 20587 raft_consensus.cc:348] T 3bcf20135d654c72ac0ef54d4cb4ef19 P 3f521b440d444258979d80243563fecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49042bd14de4c10a2497b53319da7a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } } peers { permanent_uuid: "3f521b440d444258979d80243563fecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33531 } } peers { permanent_uuid: "b0c4ca74cab04690a326001e8d77ae5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41025 } }
I20260929 01:06:12.865661 20587 raft_consensus.cc:374] T 3bcf20135d654c72ac0ef54d4cb4ef19 P 3f521b440d444258979d80243563fecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:12.865720 20587 raft_consensus.cc:729] T 3bcf20135d654c72ac0ef54d4cb4ef19 P 3f521b440d444258979d80243563fecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f521b440d444258979d80243563fecf, State: Initialized, Role: FOLLOWER
I20260929 01:06:12.865696 20586 consensus_queue.cc:260] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49042bd14de4c10a2497b53319da7a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } } peers { permanent_uuid: "3f521b440d444258979d80243563fecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33531 } } peers { permanent_uuid: "b0c4ca74cab04690a326001e8d77ae5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41025 } }
I20260929 01:06:12.866122 20587 consensus_queue.cc:260] T 3bcf20135d654c72ac0ef54d4cb4ef19 P 3f521b440d444258979d80243563fecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49042bd14de4c10a2497b53319da7a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } } peers { permanent_uuid: "3f521b440d444258979d80243563fecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33531 } } peers { permanent_uuid: "b0c4ca74cab04690a326001e8d77ae5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41025 } }
I20260929 01:06:12.866415 20569 heartbeater.cc:503] Master 127.0.98.62:46511 was elected leader, sending a full tablet report...
I20260929 01:06:12.866672 20586 ts_tablet_manager.cc:1428] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 01:06:12.867029 20587 ts_tablet_manager.cc:1428] T 3bcf20135d654c72ac0ef54d4cb4ef19 P 3f521b440d444258979d80243563fecf: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260929 01:06:12.867332 20307 heartbeater.cc:503] Master 127.0.98.62:46511 was elected leader, sending a full tablet report...
I20260929 01:06:12.868366 20588 raft_consensus.cc:348] T 3bcf20135d654c72ac0ef54d4cb4ef19 P d49042bd14de4c10a2497b53319da7a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49042bd14de4c10a2497b53319da7a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } } peers { permanent_uuid: "3f521b440d444258979d80243563fecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33531 } } peers { permanent_uuid: "b0c4ca74cab04690a326001e8d77ae5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41025 } }
I20260929 01:06:12.868594 20588 raft_consensus.cc:374] T 3bcf20135d654c72ac0ef54d4cb4ef19 P d49042bd14de4c10a2497b53319da7a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:12.868669 20588 raft_consensus.cc:729] T 3bcf20135d654c72ac0ef54d4cb4ef19 P d49042bd14de4c10a2497b53319da7a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d49042bd14de4c10a2497b53319da7a3, State: Initialized, Role: FOLLOWER
I20260929 01:06:12.869077 20588 consensus_queue.cc:260] T 3bcf20135d654c72ac0ef54d4cb4ef19 P d49042bd14de4c10a2497b53319da7a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49042bd14de4c10a2497b53319da7a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } } peers { permanent_uuid: "3f521b440d444258979d80243563fecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33531 } } peers { permanent_uuid: "b0c4ca74cab04690a326001e8d77ae5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41025 } }
I20260929 01:06:12.869837 20438 heartbeater.cc:503] Master 127.0.98.62:46511 was elected leader, sending a full tablet report...
I20260929 01:06:12.869999 20588 ts_tablet_manager.cc:1428] T 3bcf20135d654c72ac0ef54d4cb4ef19 P d49042bd14de4c10a2497b53319da7a3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 01:06:12.874889 20592 raft_consensus.cc:482] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:06:12.875018 20592 raft_consensus.cc:504] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49042bd14de4c10a2497b53319da7a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } } peers { permanent_uuid: "3f521b440d444258979d80243563fecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33531 } } peers { permanent_uuid: "b0c4ca74cab04690a326001e8d77ae5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41025 } }
I20260929 01:06:12.875877 20592 leader_election.cc:288] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d49042bd14de4c10a2497b53319da7a3 (127.0.98.2:34115), 3f521b440d444258979d80243563fecf (127.0.98.1:33531)
I20260929 01:06:12.879195 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:12.876151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50723 (local address 127.0.98.2:34115)
0929 01:06:12.876283 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:12.876287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:12.876303 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:06:12.876357 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:12.876360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:12.876410 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:12.876486 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:12.876759 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.877247 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.878259 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.878651 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.878763 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:12.878840 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:12.879038 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 01:06:12.879108 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 01:06:12.879204 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:12.876085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34115 (local address 127.0.98.3:50723)
0929 01:06:12.876186 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:12.876203 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:12.876281 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:12.876496 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:12.876499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:12.876515 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:12.876632 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:12.876641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.877355 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.877360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:12.878138 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:12.878146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.878763 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.878766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:12.878904 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:12.878928 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 01:06:12.879007 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 01:06:12.879073 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":1}
I20260929 01:06:12.879839 20392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcf20135d654c72ac0ef54d4cb4ef19" candidate_uuid: "b0c4ca74cab04690a326001e8d77ae5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49042bd14de4c10a2497b53319da7a3" is_pre_election: true
I20260929 01:06:12.880020 20585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:12.876656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48347 (local address 127.0.98.1:33531)
0929 01:06:12.876786 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:12.876793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:12.876807 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:06:12.876949 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:12.876962 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:12.877008 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:12.877087 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 01:06:12.877340 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.877823 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.878906 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.879460 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.879594 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:12.879648 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:12.879857 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 01:06:12.879923 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 01:06:12.880040 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:12.876323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33531 (local address 127.0.98.3:48347)
0929 01:06:12.876746 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:12.876759 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:12.876843 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:12.877097 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:12.877100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:12.877115 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 01:06:12.877237 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:12.877242 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.877931 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.877938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:12.878732 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:12.878745 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.879613 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.879617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:12.879741 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:12.879757 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 01:06:12.879842 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 01:06:12.879903 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":101,"threads_started":1}
I20260929 01:06:12.880136 20392 raft_consensus.cc:2454] T 3bcf20135d654c72ac0ef54d4cb4ef19 P d49042bd14de4c10a2497b53319da7a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0c4ca74cab04690a326001e8d77ae5d in term 0.
I20260929 01:06:12.880565 20458 leader_election.cc:302] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0c4ca74cab04690a326001e8d77ae5d, d49042bd14de4c10a2497b53319da7a3; no voters:
I20260929 01:06:12.880554 20261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcf20135d654c72ac0ef54d4cb4ef19" candidate_uuid: "b0c4ca74cab04690a326001e8d77ae5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f521b440d444258979d80243563fecf" is_pre_election: true
I20260929 01:06:12.880833 20261 raft_consensus.cc:2454] T 3bcf20135d654c72ac0ef54d4cb4ef19 P 3f521b440d444258979d80243563fecf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0c4ca74cab04690a326001e8d77ae5d in term 0.
I20260929 01:06:12.880846 20592 raft_consensus.cc:2781] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:06:12.880927 20592 raft_consensus.cc:482] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:06:12.880985 20592 raft_consensus.cc:3037] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:12.882019 20592 raft_consensus.cc:504] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49042bd14de4c10a2497b53319da7a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } } peers { permanent_uuid: "3f521b440d444258979d80243563fecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33531 } } peers { permanent_uuid: "b0c4ca74cab04690a326001e8d77ae5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41025 } }
I20260929 01:06:12.882349 20592 leader_election.cc:288] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d [CANDIDATE]: Term 1 election: Requested vote from peers d49042bd14de4c10a2497b53319da7a3 (127.0.98.2:34115), 3f521b440d444258979d80243563fecf (127.0.98.1:33531)
I20260929 01:06:12.882715 20392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcf20135d654c72ac0ef54d4cb4ef19" candidate_uuid: "b0c4ca74cab04690a326001e8d77ae5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49042bd14de4c10a2497b53319da7a3"
I20260929 01:06:12.882782 20261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcf20135d654c72ac0ef54d4cb4ef19" candidate_uuid: "b0c4ca74cab04690a326001e8d77ae5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f521b440d444258979d80243563fecf"
I20260929 01:06:12.882875 20392 raft_consensus.cc:3037] T 3bcf20135d654c72ac0ef54d4cb4ef19 P d49042bd14de4c10a2497b53319da7a3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:12.882956 20261 raft_consensus.cc:3037] T 3bcf20135d654c72ac0ef54d4cb4ef19 P 3f521b440d444258979d80243563fecf [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:12.883862 20392 raft_consensus.cc:2454] T 3bcf20135d654c72ac0ef54d4cb4ef19 P d49042bd14de4c10a2497b53319da7a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0c4ca74cab04690a326001e8d77ae5d in term 1.
I20260929 01:06:12.883946 20261 raft_consensus.cc:2454] T 3bcf20135d654c72ac0ef54d4cb4ef19 P 3f521b440d444258979d80243563fecf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0c4ca74cab04690a326001e8d77ae5d in term 1.
I20260929 01:06:12.884166 20458 leader_election.cc:302] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0c4ca74cab04690a326001e8d77ae5d, d49042bd14de4c10a2497b53319da7a3; no voters:
I20260929 01:06:12.884434 20592 raft_consensus.cc:2781] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:12.884666 20592 raft_consensus.cc:686] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d [term 1 LEADER]: Becoming Leader. State: Replica: b0c4ca74cab04690a326001e8d77ae5d, State: Running, Role: LEADER
I20260929 01:06:12.884972 20592 consensus_queue.cc:237] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49042bd14de4c10a2497b53319da7a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } } peers { permanent_uuid: "3f521b440d444258979d80243563fecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33531 } } peers { permanent_uuid: "b0c4ca74cab04690a326001e8d77ae5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41025 } }
I20260929 01:06:12.887975 20119 catalog_manager.cc:5495] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d reported cstate change: term changed from 0 to 1, leader changed from <none> to b0c4ca74cab04690a326001e8d77ae5d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b0c4ca74cab04690a326001e8d77ae5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d49042bd14de4c10a2497b53319da7a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f521b440d444258979d80243563fecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0c4ca74cab04690a326001e8d77ae5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41025 } health_report { overall_health: HEALTHY } } }
W20260929 01:06:12.897998 20308 tablet.cc:2369] T 3bcf20135d654c72ac0ef54d4cb4ef19 P 3f521b440d444258979d80243563fecf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:06:12.907565 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:12.904055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49356 (local address 127.0.98.3:41025)
0929 01:06:12.904231 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:12.904238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:12.904255 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:06:12.904343 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:12.904346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:12.904395 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:12.904509 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:12.904753 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.905257 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.906103 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:12.906330 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:12.906440 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:12.906740 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:12.907193 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:12.907319 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:12.907379 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 01:06:12.907425 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 01:06:12.913594 20261 raft_consensus.cc:1261] T 3bcf20135d654c72ac0ef54d4cb4ef19 P 3f521b440d444258979d80243563fecf [term 1 FOLLOWER]: Refusing update from remote peer b0c4ca74cab04690a326001e8d77ae5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:06:12.913648 20392 raft_consensus.cc:1261] T 3bcf20135d654c72ac0ef54d4cb4ef19 P d49042bd14de4c10a2497b53319da7a3 [term 1 FOLLOWER]: Refusing update from remote peer b0c4ca74cab04690a326001e8d77ae5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:06:12.914312 20592 consensus_queue.cc:1035] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f521b440d444258979d80243563fecf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:06:12.914527 20596 consensus_queue.cc:1035] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d49042bd14de4c10a2497b53319da7a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:06:12.922518 20601 mvcc.cc:205] Tried to move back new op lower bound from 7334477713046667264 to 7334477712939335680. Current Snapshot: MvccSnapshot[applied={T|T < 7334477713046667264}]
I20260929 01:06:12.929697 20119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:06:12.929831 20119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 01:06:12.932096 20119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d384c50beb44507a82ca6ecf95c8011: Sending DeleteTablet for 3 replicas of tablet 3bcf20135d654c72ac0ef54d4cb4ef19
I20260929 01:06:12.932807 20372 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bcf20135d654c72ac0ef54d4cb4ef19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:06:12 UTC) from {username='slave'} at 127.0.0.1:53150
I20260929 01:06:12.932821 20241 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bcf20135d654c72ac0ef54d4cb4ef19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:06:12 UTC) from {username='slave'} at 127.0.0.1:50138
I20260929 01:06:12.933259 20609 tablet_replica.cc:330] T 3bcf20135d654c72ac0ef54d4cb4ef19 P d49042bd14de4c10a2497b53319da7a3: stopping tablet replica
I20260929 01:06:12.933285 20503 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bcf20135d654c72ac0ef54d4cb4ef19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 01:06:12 UTC) from {username='slave'} at 127.0.0.1:49344
I20260929 01:06:12.933559 20609 raft_consensus.cc:2229] T 3bcf20135d654c72ac0ef54d4cb4ef19 P d49042bd14de4c10a2497b53319da7a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 01:06:12.933735 20611 tablet_replica.cc:330] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d: stopping tablet replica
I20260929 01:06:12.933848 20609 raft_consensus.cc:2258] T 3bcf20135d654c72ac0ef54d4cb4ef19 P d49042bd14de4c10a2497b53319da7a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:06:12.934092 20611 raft_consensus.cc:2229] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d [term 1 LEADER]: Raft consensus shutting down.
I20260929 01:06:12.934502 20611 raft_consensus.cc:2258] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 01:06:12.934544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 20179
I20260929 01:06:12.935597 20609 ts_tablet_manager.cc:1905] T 3bcf20135d654c72ac0ef54d4cb4ef19 P d49042bd14de4c10a2497b53319da7a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:06:12.936654 20611 ts_tablet_manager.cc:1905] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:06:12.938791 20609 ts_tablet_manager.cc:1918] T 3bcf20135d654c72ac0ef54d4cb4ef19 P d49042bd14de4c10a2497b53319da7a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:06:12.938925 20609 log.cc:1196] T 3bcf20135d654c72ac0ef54d4cb4ef19 P d49042bd14de4c10a2497b53319da7a3: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-1/wal/wals/3bcf20135d654c72ac0ef54d4cb4ef19
I20260929 01:06:12.939258 20609 ts_tablet_manager.cc:1939] T 3bcf20135d654c72ac0ef54d4cb4ef19 P d49042bd14de4c10a2497b53319da7a3: Deleting consensus metadata
I20260929 01:06:12.939579 20611 ts_tablet_manager.cc:1918] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 01:06:12.939718 20611 log.cc:1196] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d: Deleting WAL directory at /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790643720886468-392-0/minicluster-data/ts-2/wal/wals/3bcf20135d654c72ac0ef54d4cb4ef19
I20260929 01:06:12.940060 20611 ts_tablet_manager.cc:1939] T 3bcf20135d654c72ac0ef54d4cb4ef19 P b0c4ca74cab04690a326001e8d77ae5d: Deleting consensus metadata
I20260929 01:06:12.940454 20105 catalog_manager.cc:4841] TS d49042bd14de4c10a2497b53319da7a3 (127.0.98.2:34115): tablet 3bcf20135d654c72ac0ef54d4cb4ef19 (table test-table [id=d44e9e6f4b934e03ba2a68fd63618b9d]) successfully deleted
I20260929 01:06:12.941186 20106 catalog_manager.cc:4841] TS b0c4ca74cab04690a326001e8d77ae5d (127.0.98.3:41025): tablet 3bcf20135d654c72ac0ef54d4cb4ef19 (table test-table [id=d44e9e6f4b934e03ba2a68fd63618b9d]) successfully deleted
W20260929 01:06:12.943673 20105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33531 (error 108)
I20260929 01:06:12.943915 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 20310
I20260929 01:06:12.950836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 20441
I20260929 01:06:12.951637 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:12.951009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33531 (local address 127.0.0.1:50142)
0929 01:06:12.951480 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:12.951550 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391}
W20260929 01:06:12.951844 20105 catalog_manager.cc:4584] TS 3f521b440d444258979d80243563fecf (127.0.98.1:33531): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3bcf20135d654c72ac0ef54d4cb4ef19: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33531: connect: Connection refused (error 111)
I20260929 01:06:12.957118 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 20088
2026-09-29T01:06:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1023 ms)
[----------] 6 tests from EncryptionPolicyTest (9939 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:13 dist-test-slave-7zdl krb5kdc[20615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:13 dist-test-slave-7zdl krb5kdc[20615](info): set up 2 sockets
Sep 29 01:06:13 dist-test-slave-7zdl krb5kdc[20615](info): commencing operation
krb5kdc: starting...
W20260929 01:06:13.991135 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:06:14 dist-test-slave-7zdl krb5kdc[20615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:06:14.113379 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36553
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:14.223295 20628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:14.223546 20628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:14.223587 20628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:14.227522 20628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:14.227576 20628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:14.227595 20628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:14.227612 20628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:14.227626 20628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:14.232846 20628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36553
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:14.234185 20628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:14.235251 20628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:14.243606 20633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:14.243667 20628 server_base.cc:1034] running on GCE node
W20260929 01:06:14.243600 20636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:14.243600 20634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:14.244426 20628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:14.245435 20628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:14.246645 20628 hybrid_clock.cc:648] HybridClock initialized: now 1790643974246613 us; error 44 us; skew 500 ppm
Sep 29 01:06:14 dist-test-slave-7zdl krb5kdc[20615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:14.254429 20628 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:06:14.255472 20628 webserver.cc:460] Webserver started at http://127.0.0.1:37497/ using document root <none> and password file <none>
I20260929 01:06:14.256031 20628 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:14.256088 20628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:14.256325 20628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:14.258669 20628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "de8757871005478882f6df79052cf5e0"
format_stamp: "Formatted at 2026-09-29 01:06:14 on dist-test-slave-7zdl"
server_key: "e7914d9abaeff5d994c0408aa2d69ca9"
server_key_iv: "b49e5e6068581b301fe7608077a79c36"
server_key_version: "encryptionkey@0"
I20260929 01:06:14.259290 20628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "de8757871005478882f6df79052cf5e0"
format_stamp: "Formatted at 2026-09-29 01:06:14 on dist-test-slave-7zdl"
server_key: "e7914d9abaeff5d994c0408aa2d69ca9"
server_key_iv: "b49e5e6068581b301fe7608077a79c36"
server_key_version: "encryptionkey@0"
I20260929 01:06:14.262929 20628 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:06:14.265326 20643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:14.266588 20628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:06:14.266726 20628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "de8757871005478882f6df79052cf5e0"
format_stamp: "Formatted at 2026-09-29 01:06:14 on dist-test-slave-7zdl"
server_key: "e7914d9abaeff5d994c0408aa2d69ca9"
server_key_iv: "b49e5e6068581b301fe7608077a79c36"
server_key_version: "encryptionkey@0"
I20260929 01:06:14.266867 20628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:14.281836 20628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:14.285079 20628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:14.285272 20628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:14.292377 20628 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36553
I20260929 01:06:14.292397 20695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36553 every 8 connection(s)
I20260929 01:06:14.293464 20628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:14.296092 20696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:14.299127 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 20628
I20260929 01:06:14.299268 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:14.299571 392 external_mini_cluster.cc:1428] Setting key cdbb67b090c5dff3beea6aa088fcb683
I20260929 01:06:14.302001 20696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de8757871005478882f6df79052cf5e0: Bootstrap starting.
I20260929 01:06:14.304642 20696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de8757871005478882f6df79052cf5e0: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:14.305480 20696 log.cc:824] T 00000000000000000000000000000000 P de8757871005478882f6df79052cf5e0: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:14.307660 20696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de8757871005478882f6df79052cf5e0: No bootstrap required, opened a new log
Sep 29 01:06:14 dist-test-slave-7zdl krb5kdc[20615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 01:06:14.311499 20696 raft_consensus.cc:348] T 00000000000000000000000000000000 P de8757871005478882f6df79052cf5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de8757871005478882f6df79052cf5e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } }
I20260929 01:06:14.311713 20696 raft_consensus.cc:374] T 00000000000000000000000000000000 P de8757871005478882f6df79052cf5e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:14.311784 20696 raft_consensus.cc:729] T 00000000000000000000000000000000 P de8757871005478882f6df79052cf5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de8757871005478882f6df79052cf5e0, State: Initialized, Role: FOLLOWER
I20260929 01:06:14.312199 20696 consensus_queue.cc:260] T 00000000000000000000000000000000 P de8757871005478882f6df79052cf5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de8757871005478882f6df79052cf5e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } }
I20260929 01:06:14.312346 20696 raft_consensus.cc:388] T 00000000000000000000000000000000 P de8757871005478882f6df79052cf5e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:14.312417 20696 raft_consensus.cc:482] T 00000000000000000000000000000000 P de8757871005478882f6df79052cf5e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:14.312529 20696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de8757871005478882f6df79052cf5e0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:14.313571 20696 raft_consensus.cc:504] T 00000000000000000000000000000000 P de8757871005478882f6df79052cf5e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de8757871005478882f6df79052cf5e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } }
I20260929 01:06:14.313966 20696 leader_election.cc:302] T 00000000000000000000000000000000 P de8757871005478882f6df79052cf5e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de8757871005478882f6df79052cf5e0; no voters:
I20260929 01:06:14.314307 20696 leader_election.cc:288] T 00000000000000000000000000000000 P de8757871005478882f6df79052cf5e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:14.314683 20702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de8757871005478882f6df79052cf5e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:14.314940 20702 raft_consensus.cc:686] T 00000000000000000000000000000000 P de8757871005478882f6df79052cf5e0 [term 1 LEADER]: Becoming Leader. State: Replica: de8757871005478882f6df79052cf5e0, State: Running, Role: LEADER
I20260929 01:06:14.315263 20702 consensus_queue.cc:237] T 00000000000000000000000000000000 P de8757871005478882f6df79052cf5e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de8757871005478882f6df79052cf5e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } }
I20260929 01:06:14.315642 20696 sys_catalog.cc:564] T 00000000000000000000000000000000 P de8757871005478882f6df79052cf5e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:14.315714 20699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:14.300999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53769 (local address 127.0.0.1:36553)
0929 01:06:14.301420 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:14.301439 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:14.301480 (+ 41us) server_negotiation.cc:407] Connection header received
0929 01:06:14.302288 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:14.302311 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:14.302748 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:14.303138 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:14.303787 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:14.305281 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:14.306380 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:14.307051 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:14.307212 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:14.309925 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:14.309972 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:14.309984 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:14.310010 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:14.312204 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:14.312791 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:14.312796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:14.312800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:14.312925 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:14.313296 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:14.313298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:14.313299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:14.313788 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:14.313965 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:14.314385 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 01:06:14.314638 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":116,"threads_started":1}
I20260929 01:06:14.316533 20702 sys_catalog.cc:455] T 00000000000000000000000000000000 P de8757871005478882f6df79052cf5e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de8757871005478882f6df79052cf5e0. Latest consensus state: current_term: 1 leader_uuid: "de8757871005478882f6df79052cf5e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de8757871005478882f6df79052cf5e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } } }
I20260929 01:06:14.316648 20702 sys_catalog.cc:458] T 00000000000000000000000000000000 P de8757871005478882f6df79052cf5e0 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:14.316962 20703 sys_catalog.cc:455] T 00000000000000000000000000000000 P de8757871005478882f6df79052cf5e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de8757871005478882f6df79052cf5e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de8757871005478882f6df79052cf5e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36553 } } }
I20260929 01:06:14.317046 20703 sys_catalog.cc:458] T 00000000000000000000000000000000 P de8757871005478882f6df79052cf5e0 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:14.319118 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 01:06:14.321044 20716 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de8757871005478882f6df79052cf5e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:06:14.321125 20716 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:06:14.321185 20710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:14.322079 20710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:14.328320 20710 catalog_manager.cc:1347] Generated new cluster ID: c7355b18e3fb40fd8b085c37c036b3ab
I20260929 01:06:14.328387 20710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:14.329746 20699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:14.321262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55693 (local address 127.0.0.1:36553)
0929 01:06:14.321373 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:14.321378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:14.321580 (+ 202us) server_negotiation.cc:407] Connection header received
0929 01:06:14.321697 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:14.321701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:14.321753 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:14.321852 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:14.322653 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:14.323459 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:14.324370 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:14.324630 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:14.324742 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:14.326169 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:14.326172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:14.326175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:14.326198 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:14.327912 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:14.328473 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:14.328477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:14.328480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:14.328567 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:14.328881 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:14.328884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:14.328886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:14.329058 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:14.329213 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:14.329466 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 01:06:14.329555 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 01:06:14.341614 20710 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:14.342887 20710 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:14.355911 20710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de8757871005478882f6df79052cf5e0: Generated new TSK 0
I20260929 01:06:14.356586 20710 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:14.363854 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 20628
W20260929 01:06:14.369285 20613 connection.cc:535] client connection to 127.0.0.1:36553 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36553): Connection reset by peer (error 104)
2026-09-29T01:06:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:14 dist-test-slave-7zdl krb5kdc[20727](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:14 dist-test-slave-7zdl krb5kdc[20727](info): set up 2 sockets
Sep 29 01:06:14 dist-test-slave-7zdl krb5kdc[20727](info): commencing operation
krb5kdc: starting...
W20260929 01:06:15.391155 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:06:15 dist-test-slave-7zdl krb5kdc[20727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:06:15.513697 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35239
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:15.622215 20740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:15.622500 20740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:15.622584 20740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:15.626621 20740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:15.626706 20740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:15.626744 20740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:15.626781 20740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:15.626833 20740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:15.632140 20740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35239
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:15.633282 20740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:15.634349 20740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:15.643169 20748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:15.643134 20745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:15.643226 20740 server_base.cc:1034] running on GCE node
W20260929 01:06:15.643139 20746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:15.643900 20740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:15.645080 20740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:15.646291 20740 hybrid_clock.cc:648] HybridClock initialized: now 1790643975646260 us; error 46 us; skew 500 ppm
Sep 29 01:06:15 dist-test-slave-7zdl krb5kdc[20727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:15.649201 20740 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:06:15.650130 20740 webserver.cc:460] Webserver started at http://127.0.0.1:41833/ using document root <none> and password file <none>
I20260929 01:06:15.650694 20740 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:15.650795 20740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:15.651053 20740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:15.653023 20740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "1050b549cbdc4da2b3a31573f3d3644d"
format_stamp: "Formatted at 2026-09-29 01:06:15 on dist-test-slave-7zdl"
server_key: "c34ef4c0b481f8a99686b36cc31bdaa9"
server_key_iv: "e3df1aace4ad0e99130175f774f05215"
server_key_version: "encryptionkey@0"
I20260929 01:06:15.653584 20740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "1050b549cbdc4da2b3a31573f3d3644d"
format_stamp: "Formatted at 2026-09-29 01:06:15 on dist-test-slave-7zdl"
server_key: "c34ef4c0b481f8a99686b36cc31bdaa9"
server_key_iv: "e3df1aace4ad0e99130175f774f05215"
server_key_version: "encryptionkey@0"
I20260929 01:06:15.657289 20740 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:06:15.659742 20755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:15.661031 20740 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:06:15.661170 20740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "1050b549cbdc4da2b3a31573f3d3644d"
format_stamp: "Formatted at 2026-09-29 01:06:15 on dist-test-slave-7zdl"
server_key: "c34ef4c0b481f8a99686b36cc31bdaa9"
server_key_iv: "e3df1aace4ad0e99130175f774f05215"
server_key_version: "encryptionkey@0"
I20260929 01:06:15.661314 20740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:15.691084 20740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:15.694552 20740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:15.694766 20740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:15.702138 20740 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35239
I20260929 01:06:15.702164 20807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35239 every 8 connection(s)
I20260929 01:06:15.703118 20740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:15.706246 20808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:15.709592 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 20740
I20260929 01:06:15.709741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:15.710089 392 external_mini_cluster.cc:1428] Setting key e964deea9eabd283bcac9946e931f083
I20260929 01:06:15.713833 20808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1050b549cbdc4da2b3a31573f3d3644d: Bootstrap starting.
I20260929 01:06:15.716567 20808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1050b549cbdc4da2b3a31573f3d3644d: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:15.717481 20808 log.cc:824] T 00000000000000000000000000000000 P 1050b549cbdc4da2b3a31573f3d3644d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:15.721884 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:15.711741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54673 (local address 127.0.0.1:35239)
0929 01:06:15.712250 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:15.712269 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:15.712479 (+ 210us) server_negotiation.cc:407] Connection header received
0929 01:06:15.713390 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:15.713411 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:15.713826 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:15.714191 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:15.715004 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:15.716481 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:15.717733 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:15.718413 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:15.718551 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:15.719773 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:15.719779 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:15.719796 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:15.720288 (+ 492us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:15.720351 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:15.720379 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:15.720492 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:15.720852 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 01:06:15.720986 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":151,"threads_started":1}
I20260929 01:06:15.724079 20808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1050b549cbdc4da2b3a31573f3d3644d: No bootstrap required, opened a new log
I20260929 01:06:15.725152 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:15.727679 20808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1050b549cbdc4da2b3a31573f3d3644d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1050b549cbdc4da2b3a31573f3d3644d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35239 } }
I20260929 01:06:15.727977 20808 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1050b549cbdc4da2b3a31573f3d3644d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:15.728050 20808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1050b549cbdc4da2b3a31573f3d3644d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1050b549cbdc4da2b3a31573f3d3644d, State: Initialized, Role: FOLLOWER
I20260929 01:06:15.728485 20808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1050b549cbdc4da2b3a31573f3d3644d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1050b549cbdc4da2b3a31573f3d3644d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35239 } }
I20260929 01:06:15.728631 20808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1050b549cbdc4da2b3a31573f3d3644d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:15.728711 20808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1050b549cbdc4da2b3a31573f3d3644d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:15.728822 20808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1050b549cbdc4da2b3a31573f3d3644d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:15.729990 20808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1050b549cbdc4da2b3a31573f3d3644d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1050b549cbdc4da2b3a31573f3d3644d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35239 } }
I20260929 01:06:15.730345 20808 leader_election.cc:302] T 00000000000000000000000000000000 P 1050b549cbdc4da2b3a31573f3d3644d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1050b549cbdc4da2b3a31573f3d3644d; no voters:
I20260929 01:06:15.730681 20808 leader_election.cc:288] T 00000000000000000000000000000000 P 1050b549cbdc4da2b3a31573f3d3644d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:15.730788 20818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1050b549cbdc4da2b3a31573f3d3644d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:15.731102 20818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1050b549cbdc4da2b3a31573f3d3644d [term 1 LEADER]: Becoming Leader. State: Replica: 1050b549cbdc4da2b3a31573f3d3644d, State: Running, Role: LEADER
I20260929 01:06:15.731473 20818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1050b549cbdc4da2b3a31573f3d3644d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1050b549cbdc4da2b3a31573f3d3644d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35239 } }
I20260929 01:06:15.732035 20808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1050b549cbdc4da2b3a31573f3d3644d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:15.732424 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:15.727294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53397 (local address 127.0.0.1:35239)
0929 01:06:15.727421 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:15.727426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:15.727593 (+ 167us) server_negotiation.cc:407] Connection header received
0929 01:06:15.727699 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:15.727701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:15.727735 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:15.727799 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:15.728086 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:15.728818 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:15.730002 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:15.730308 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:15.730387 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:15.731816 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:15.731819 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:15.731822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:15.731833 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:15.731858 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:15.731870 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:15.731945 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:15.732251 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 01:06:15.732285 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 01:06:15.733317 20819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1050b549cbdc4da2b3a31573f3d3644d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1050b549cbdc4da2b3a31573f3d3644d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1050b549cbdc4da2b3a31573f3d3644d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35239 } } }
I20260929 01:06:15.733438 20819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1050b549cbdc4da2b3a31573f3d3644d [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:15.733840 20827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:15.733973 20820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1050b549cbdc4da2b3a31573f3d3644d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1050b549cbdc4da2b3a31573f3d3644d. Latest consensus state: current_term: 1 leader_uuid: "1050b549cbdc4da2b3a31573f3d3644d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1050b549cbdc4da2b3a31573f3d3644d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35239 } } }
I20260929 01:06:15.734064 20820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1050b549cbdc4da2b3a31573f3d3644d [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:15.736959 20827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:15.742128 20827 catalog_manager.cc:1347] Generated new cluster ID: ef578a9e3c234f2b913bdaaeb0b79a7b
I20260929 01:06:15.742193 20827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:15.759867 20827 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:15.760792 20827 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:15.765421 20827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1050b549cbdc4da2b3a31573f3d3644d: Generated new TSK 0
I20260929 01:06:15.766104 20827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:15.774899 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 20740
W20260929 01:06:15.779621 20725 connection.cc:535] client connection to 127.0.0.1:35239 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35239): Connection reset by peer (error 104)
2026-09-29T01:06:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:15 dist-test-slave-7zdl krb5kdc[20839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:15 dist-test-slave-7zdl krb5kdc[20839](info): set up 2 sockets
Sep 29 01:06:15 dist-test-slave-7zdl krb5kdc[20839](info): commencing operation
krb5kdc: starting...
W20260929 01:06:16.811048 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:06:16 dist-test-slave-7zdl krb5kdc[20839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:06:16.930275 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36087
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:17.037246 20852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:17.037539 20852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:17.037608 20852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:17.041563 20852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:17.041651 20852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:17.041689 20852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:17.041749 20852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:17.041788 20852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:17.046934 20852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36087
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:17.047945 20852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:17.048959 20852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:17.057256 20858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:17.057287 20857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:17.057265 20860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:17.057704 20852 server_base.cc:1034] running on GCE node
I20260929 01:06:17.058226 20852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:17.059369 20852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:17.060519 20852 hybrid_clock.cc:648] HybridClock initialized: now 1790643977060499 us; error 41 us; skew 500 ppm
Sep 29 01:06:17 dist-test-slave-7zdl krb5kdc[20839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:17.063297 20852 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:06:17.064154 20852 webserver.cc:460] Webserver started at http://127.0.0.1:33033/ using document root <none> and password file <none>
I20260929 01:06:17.064688 20852 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:17.064749 20852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:17.064972 20852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:17.066991 20852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "64e6128d34f4494fb24761e0edc72429"
format_stamp: "Formatted at 2026-09-29 01:06:17 on dist-test-slave-7zdl"
server_key: "907cec43c6d60274b3c611d2e3c178f9"
server_key_iv: "bb86197efd4a8a625f54f0d03f131912"
server_key_version: "encryptionkey@0"
I20260929 01:06:17.067545 20852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "64e6128d34f4494fb24761e0edc72429"
format_stamp: "Formatted at 2026-09-29 01:06:17 on dist-test-slave-7zdl"
server_key: "907cec43c6d60274b3c611d2e3c178f9"
server_key_iv: "bb86197efd4a8a625f54f0d03f131912"
server_key_version: "encryptionkey@0"
I20260929 01:06:17.071056 20852 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:06:17.073397 20867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:17.074524 20852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:06:17.074645 20852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "64e6128d34f4494fb24761e0edc72429"
format_stamp: "Formatted at 2026-09-29 01:06:17 on dist-test-slave-7zdl"
server_key: "907cec43c6d60274b3c611d2e3c178f9"
server_key_iv: "bb86197efd4a8a625f54f0d03f131912"
server_key_version: "encryptionkey@0"
I20260929 01:06:17.074747 20852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:17.079172 20852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:17.081904 20852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:17.082098 20852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:17.088912 20852 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36087
I20260929 01:06:17.089039 20919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36087 every 8 connection(s)
I20260929 01:06:17.089860 20852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:17.092608 20920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:17.095603 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 20852
I20260929 01:06:17.095777 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:17.096056 392 external_mini_cluster.cc:1428] Setting key ba56c669ecfc285e99ec3bf8c9eb52d3
I20260929 01:06:17.098508 20920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64e6128d34f4494fb24761e0edc72429: Bootstrap starting.
I20260929 01:06:17.101105 20920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64e6128d34f4494fb24761e0edc72429: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:17.101168 20923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:17.097397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43677 (local address 127.0.0.1:36087)
0929 01:06:17.097784 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:17.097798 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:17.097832 (+ 34us) server_negotiation.cc:407] Connection header received
0929 01:06:17.098435 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:17.098451 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:17.098776 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:17.099117 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:17.099970 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:17.099975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:17.099986 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:17.099997 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:17.100046 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:17.100066 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:17.100175 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:17.100307 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 01:06:17.100468 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":128,"threads_started":1}
I20260929 01:06:17.102077 20920 log.cc:824] T 00000000000000000000000000000000 P 64e6128d34f4494fb24761e0edc72429: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:17.104059 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:17.104658 20920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64e6128d34f4494fb24761e0edc72429: No bootstrap required, opened a new log
I20260929 01:06:17.107681 20923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:17.106151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52417 (local address 127.0.0.1:36087)
0929 01:06:17.106263 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:17.106267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:17.106405 (+ 138us) server_negotiation.cc:407] Connection header received
0929 01:06:17.106523 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:17.106525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:17.106551 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:17.106608 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:17.107304 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:17.107307 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:17.107309 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:17.107318 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:17.107340 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:17.107350 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:17.107415 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:17.107526 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 01:06:17.107553 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 01:06:17.108946 20920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64e6128d34f4494fb24761e0edc72429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e6128d34f4494fb24761e0edc72429" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36087 } }
I20260929 01:06:17.109154 20920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64e6128d34f4494fb24761e0edc72429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:17.109227 20920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64e6128d34f4494fb24761e0edc72429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64e6128d34f4494fb24761e0edc72429, State: Initialized, Role: FOLLOWER
I20260929 01:06:17.109615 20920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64e6128d34f4494fb24761e0edc72429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e6128d34f4494fb24761e0edc72429" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36087 } }
I20260929 01:06:17.109752 20920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64e6128d34f4494fb24761e0edc72429 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:17.109817 20920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64e6128d34f4494fb24761e0edc72429 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:17.109912 20920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64e6128d34f4494fb24761e0edc72429 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:17.110960 20920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64e6128d34f4494fb24761e0edc72429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e6128d34f4494fb24761e0edc72429" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36087 } }
I20260929 01:06:17.111246 20920 leader_election.cc:302] T 00000000000000000000000000000000 P 64e6128d34f4494fb24761e0edc72429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64e6128d34f4494fb24761e0edc72429; no voters:
I20260929 01:06:17.111480 20920 leader_election.cc:288] T 00000000000000000000000000000000 P 64e6128d34f4494fb24761e0edc72429 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:17.111610 20930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64e6128d34f4494fb24761e0edc72429 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:17.111871 20930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64e6128d34f4494fb24761e0edc72429 [term 1 LEADER]: Becoming Leader. State: Replica: 64e6128d34f4494fb24761e0edc72429, State: Running, Role: LEADER
I20260929 01:06:17.112166 20930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64e6128d34f4494fb24761e0edc72429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e6128d34f4494fb24761e0edc72429" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36087 } }
I20260929 01:06:17.112639 20920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64e6128d34f4494fb24761e0edc72429 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:17.113723 20932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64e6128d34f4494fb24761e0edc72429 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64e6128d34f4494fb24761e0edc72429. Latest consensus state: current_term: 1 leader_uuid: "64e6128d34f4494fb24761e0edc72429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e6128d34f4494fb24761e0edc72429" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36087 } } }
I20260929 01:06:17.113839 20932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64e6128d34f4494fb24761e0edc72429 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:17.114192 20939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:17.114171 20931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64e6128d34f4494fb24761e0edc72429 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64e6128d34f4494fb24761e0edc72429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e6128d34f4494fb24761e0edc72429" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36087 } } }
I20260929 01:06:17.114928 20931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64e6128d34f4494fb24761e0edc72429 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:17.117075 20939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:17.121956 20939 catalog_manager.cc:1347] Generated new cluster ID: 9d65d24c574f46f1bd3b7283c87ca25f
I20260929 01:06:17.122031 20939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:17.138079 20939 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:17.138406 20939 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:17.147192 20939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64e6128d34f4494fb24761e0edc72429: Generated new TSK 0
I20260929 01:06:17.147917 20939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:17.157205 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 20852
2026-09-29T01:06:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:17 dist-test-slave-7zdl krb5kdc[20951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:17 dist-test-slave-7zdl krb5kdc[20951](info): set up 2 sockets
Sep 29 01:06:17 dist-test-slave-7zdl krb5kdc[20951](info): commencing operation
krb5kdc: starting...
W20260929 01:06:18.191045 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:06:18 dist-test-slave-7zdl krb5kdc[20951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:06:18.310305 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35189
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:18.416922 20964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:18.417198 20964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:18.417238 20964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:18.421243 20964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:18.421296 20964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:18.421315 20964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:18.421332 20964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:18.421346 20964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:18.426522 20964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:18.427556 20964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:18.428551 20964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:18.437073 20970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:18.437111 20969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:18.437104 20972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:18.437258 20964 server_base.cc:1034] running on GCE node
I20260929 01:06:18.437847 20964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:18.438987 20964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:18.440162 20964 hybrid_clock.cc:648] HybridClock initialized: now 1790643978440149 us; error 29 us; skew 500 ppm
Sep 29 01:06:18 dist-test-slave-7zdl krb5kdc[20951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:18.442870 20964 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:06:18.443706 20964 webserver.cc:460] Webserver started at http://127.0.0.1:39235/ using document root <none> and password file <none>
I20260929 01:06:18.444257 20964 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:18.444347 20964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:18.444581 20964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:18.446537 20964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "950a993c453446d3af56d2bdeb0e3fd8"
format_stamp: "Formatted at 2026-09-29 01:06:18 on dist-test-slave-7zdl"
server_key: "677bc2d82117bb8a30e8563b379d0cfb"
server_key_iv: "4b2c8dd5648f4f129b2575832f55bea4"
server_key_version: "encryptionkey@0"
I20260929 01:06:18.447111 20964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "950a993c453446d3af56d2bdeb0e3fd8"
format_stamp: "Formatted at 2026-09-29 01:06:18 on dist-test-slave-7zdl"
server_key: "677bc2d82117bb8a30e8563b379d0cfb"
server_key_iv: "4b2c8dd5648f4f129b2575832f55bea4"
server_key_version: "encryptionkey@0"
I20260929 01:06:18.450637 20964 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:06:18.453063 20979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:18.454277 20964 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:06:18.454416 20964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "950a993c453446d3af56d2bdeb0e3fd8"
format_stamp: "Formatted at 2026-09-29 01:06:18 on dist-test-slave-7zdl"
server_key: "677bc2d82117bb8a30e8563b379d0cfb"
server_key_iv: "4b2c8dd5648f4f129b2575832f55bea4"
server_key_version: "encryptionkey@0"
I20260929 01:06:18.454530 20964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:18.502113 20964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:18.505334 20964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:18.505533 20964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:18.512732 20964 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35189
I20260929 01:06:18.512759 21031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35189 every 8 connection(s)
I20260929 01:06:18.513638 20964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:18.516342 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 20964
I20260929 01:06:18.516502 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:18.516542 21032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:18.516878 392 external_mini_cluster.cc:1428] Setting key 4d51e8f20b3d91a01ac27c111db726d1
I20260929 01:06:18.523355 21032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 950a993c453446d3af56d2bdeb0e3fd8: Bootstrap starting.
Sep 29 01:06:18 dist-test-slave-7zdl krb5kdc[20951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 01:06:18.525660 21032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 950a993c453446d3af56d2bdeb0e3fd8: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:18.526610 21032 log.cc:824] T 00000000000000000000000000000000 P 950a993c453446d3af56d2bdeb0e3fd8: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:18.528577 21032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 950a993c453446d3af56d2bdeb0e3fd8: No bootstrap required, opened a new log
I20260929 01:06:18.530269 21035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:18.518300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49827 (local address 127.0.0.1:35189)
0929 01:06:18.518748 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:18.518761 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:18.518862 (+ 101us) server_negotiation.cc:407] Connection header received
0929 01:06:18.519472 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:18.519486 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:18.519795 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:18.520124 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:18.520914 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:18.521887 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:18.522768 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:18.523283 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:18.523460 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:18.525819 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:18.525824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:18.525836 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:18.525862 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:18.527588 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:18.528010 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:18.528020 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:18.528024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:18.528142 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:18.528424 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:18.528427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:18.528429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:18.528847 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:18.529027 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:18.529330 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 01:06:18.529598 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":170,"threads_started":1}
I20260929 01:06:18.533097 21032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 950a993c453446d3af56d2bdeb0e3fd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950a993c453446d3af56d2bdeb0e3fd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35189 } }
I20260929 01:06:18.533403 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:18.533416 21032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 950a993c453446d3af56d2bdeb0e3fd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:18.533491 21032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 950a993c453446d3af56d2bdeb0e3fd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 950a993c453446d3af56d2bdeb0e3fd8, State: Initialized, Role: FOLLOWER
I20260929 01:06:18.533955 21032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 950a993c453446d3af56d2bdeb0e3fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950a993c453446d3af56d2bdeb0e3fd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35189 } }
I20260929 01:06:18.534085 21032 raft_consensus.cc:388] T 00000000000000000000000000000000 P 950a993c453446d3af56d2bdeb0e3fd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:18.534176 21032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 950a993c453446d3af56d2bdeb0e3fd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:18.534286 21032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 950a993c453446d3af56d2bdeb0e3fd8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:18.535354 21032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 950a993c453446d3af56d2bdeb0e3fd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950a993c453446d3af56d2bdeb0e3fd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35189 } }
I20260929 01:06:18.535748 21032 leader_election.cc:302] T 00000000000000000000000000000000 P 950a993c453446d3af56d2bdeb0e3fd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 950a993c453446d3af56d2bdeb0e3fd8; no voters:
I20260929 01:06:18.536121 21032 leader_election.cc:288] T 00000000000000000000000000000000 P 950a993c453446d3af56d2bdeb0e3fd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:18.536190 21042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 950a993c453446d3af56d2bdeb0e3fd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:18.536408 21042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 950a993c453446d3af56d2bdeb0e3fd8 [term 1 LEADER]: Becoming Leader. State: Replica: 950a993c453446d3af56d2bdeb0e3fd8, State: Running, Role: LEADER
I20260929 01:06:18.536711 21042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 950a993c453446d3af56d2bdeb0e3fd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950a993c453446d3af56d2bdeb0e3fd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35189 } }
I20260929 01:06:18.537460 21032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 950a993c453446d3af56d2bdeb0e3fd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:18.538339 21043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 950a993c453446d3af56d2bdeb0e3fd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 950a993c453446d3af56d2bdeb0e3fd8. Latest consensus state: current_term: 1 leader_uuid: "950a993c453446d3af56d2bdeb0e3fd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950a993c453446d3af56d2bdeb0e3fd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35189 } } }
I20260929 01:06:18.538738 21043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 950a993c453446d3af56d2bdeb0e3fd8 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:18.538615 21044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 950a993c453446d3af56d2bdeb0e3fd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "950a993c453446d3af56d2bdeb0e3fd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950a993c453446d3af56d2bdeb0e3fd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35189 } } }
I20260929 01:06:18.539101 21044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 950a993c453446d3af56d2bdeb0e3fd8 [sys.catalog]: This master's current role is: LEADER
W20260929 01:06:18.542503 21057 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 950a993c453446d3af56d2bdeb0e3fd8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:06:18.542621 21057 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:06:18.542773 21051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:18.543823 21051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:18.544570 21035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:18.535544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36431 (local address 127.0.0.1:35189)
0929 01:06:18.535653 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:18.535657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:18.535860 (+ 203us) server_negotiation.cc:407] Connection header received
0929 01:06:18.535973 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:18.535977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:18.536018 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:18.536109 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:18.536887 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:18.537460 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:18.538689 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:18.539102 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:18.539246 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:18.541053 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:18.541057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:18.541060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:18.541086 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:18.542723 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:18.543220 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:18.543225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:18.543227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:18.543325 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:18.543656 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:18.543660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:18.543662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:18.543897 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:18.544043 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:18.544286 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 01:06:18.544411 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 01:06:18.549280 21051 catalog_manager.cc:1347] Generated new cluster ID: 2e80334fb9a44a179efa623bced8a04b
I20260929 01:06:18.549345 21051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:18.557090 21051 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:18.558254 21051 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:18.563791 21051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 950a993c453446d3af56d2bdeb0e3fd8: Generated new TSK 0
I20260929 01:06:18.564651 21051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:18.573511 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 20964
2026-09-29T01:06:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:18 dist-test-slave-7zdl krb5kdc[21063](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:18 dist-test-slave-7zdl krb5kdc[21063](info): set up 2 sockets
Sep 29 01:06:18 dist-test-slave-7zdl krb5kdc[21063](info): commencing operation
krb5kdc: starting...
W20260929 01:06:19.595096 392 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:06:19 dist-test-slave-7zdl krb5kdc[21063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:06:19.717223 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46731
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:19.826395 21076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:19.826680 21076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:19.826761 21076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:19.830785 21076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:19.830881 21076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:19.830917 21076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:19.830994 21076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:19.831041 21076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:19.836316 21076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:19.837394 21076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:19.838459 21076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:19.846985 21084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:19.847040 21082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:19.847056 21081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:19.847239 21076 server_base.cc:1034] running on GCE node
I20260929 01:06:19.847801 21076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:19.848903 21076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:19.850085 21076 hybrid_clock.cc:648] HybridClock initialized: now 1790643979850059 us; error 41 us; skew 500 ppm
Sep 29 01:06:19 dist-test-slave-7zdl krb5kdc[21063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:19.858624 21076 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:06:19.859629 21076 webserver.cc:460] Webserver started at http://127.0.0.1:38265/ using document root <none> and password file <none>
I20260929 01:06:19.860306 21076 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:19.860419 21076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:19.860662 21076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:19.862903 21076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "df6cb0dbe7cd48d094022931f6b91528"
format_stamp: "Formatted at 2026-09-29 01:06:19 on dist-test-slave-7zdl"
server_key: "48503ce2790a3bd75310572aff96b21e"
server_key_iv: "a2280ef52cb8df3f98b725f648a523c1"
server_key_version: "encryptionkey@0"
I20260929 01:06:19.863508 21076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "df6cb0dbe7cd48d094022931f6b91528"
format_stamp: "Formatted at 2026-09-29 01:06:19 on dist-test-slave-7zdl"
server_key: "48503ce2790a3bd75310572aff96b21e"
server_key_iv: "a2280ef52cb8df3f98b725f648a523c1"
server_key_version: "encryptionkey@0"
I20260929 01:06:19.867635 21076 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:06:19.870483 21091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:19.871850 21076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:06:19.872000 21076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "df6cb0dbe7cd48d094022931f6b91528"
format_stamp: "Formatted at 2026-09-29 01:06:19 on dist-test-slave-7zdl"
server_key: "48503ce2790a3bd75310572aff96b21e"
server_key_iv: "a2280ef52cb8df3f98b725f648a523c1"
server_key_version: "encryptionkey@0"
I20260929 01:06:19.872125 21076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:19.882283 21076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:19.885295 21076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:19.885489 21076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:19.892746 21076 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46731
I20260929 01:06:19.892767 21143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46731 every 8 connection(s)
I20260929 01:06:19.893796 21076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:19.896834 21144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:19.902690 21144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df6cb0dbe7cd48d094022931f6b91528: Bootstrap starting.
I20260929 01:06:19.903455 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 21076
I20260929 01:06:19.903604 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:19.903865 392 external_mini_cluster.cc:1428] Setting key 627a16c8532011fd793a7d00d5bc9834
I20260929 01:06:19.905628 21144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df6cb0dbe7cd48d094022931f6b91528: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:19.906597 21144 log.cc:824] T 00000000000000000000000000000000 P df6cb0dbe7cd48d094022931f6b91528: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:19.909409 21144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df6cb0dbe7cd48d094022931f6b91528: No bootstrap required, opened a new log
I20260929 01:06:19.914327 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:19.905174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51805 (local address 127.0.0.1:46731)
0929 01:06:19.905628 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:19.905650 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:19.905697 (+ 47us) server_negotiation.cc:407] Connection header received
0929 01:06:19.906617 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:19.906641 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:19.907137 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:19.907568 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:19.907959 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:19.909511 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:19.910389 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:19.911151 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:19.911341 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:19.912136 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:19.912143 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:19.912159 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:19.912787 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:19.912877 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:19.912919 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:19.913032 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:19.913114 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 01:06:19.913329 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":111,"threads_started":1}
I20260929 01:06:19.914377 21144 raft_consensus.cc:348] T 00000000000000000000000000000000 P df6cb0dbe7cd48d094022931f6b91528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6cb0dbe7cd48d094022931f6b91528" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46731 } }
I20260929 01:06:19.914652 21144 raft_consensus.cc:374] T 00000000000000000000000000000000 P df6cb0dbe7cd48d094022931f6b91528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:19.914718 21144 raft_consensus.cc:729] T 00000000000000000000000000000000 P df6cb0dbe7cd48d094022931f6b91528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6cb0dbe7cd48d094022931f6b91528, State: Initialized, Role: FOLLOWER
I20260929 01:06:19.915184 21144 consensus_queue.cc:260] T 00000000000000000000000000000000 P df6cb0dbe7cd48d094022931f6b91528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6cb0dbe7cd48d094022931f6b91528" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46731 } }
I20260929 01:06:19.915323 21144 raft_consensus.cc:388] T 00000000000000000000000000000000 P df6cb0dbe7cd48d094022931f6b91528 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:19.915413 21144 raft_consensus.cc:482] T 00000000000000000000000000000000 P df6cb0dbe7cd48d094022931f6b91528 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:19.915500 21144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df6cb0dbe7cd48d094022931f6b91528 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:19.916514 21144 raft_consensus.cc:504] T 00000000000000000000000000000000 P df6cb0dbe7cd48d094022931f6b91528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6cb0dbe7cd48d094022931f6b91528" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46731 } }
I20260929 01:06:19.916863 21144 leader_election.cc:302] T 00000000000000000000000000000000 P df6cb0dbe7cd48d094022931f6b91528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df6cb0dbe7cd48d094022931f6b91528; no voters:
I20260929 01:06:19.917181 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:19.917276 21144 leader_election.cc:288] T 00000000000000000000000000000000 P df6cb0dbe7cd48d094022931f6b91528 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:19.917351 21149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df6cb0dbe7cd48d094022931f6b91528 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:19.917573 21149 raft_consensus.cc:686] T 00000000000000000000000000000000 P df6cb0dbe7cd48d094022931f6b91528 [term 1 LEADER]: Becoming Leader. State: Replica: df6cb0dbe7cd48d094022931f6b91528, State: Running, Role: LEADER
I20260929 01:06:19.917881 21149 consensus_queue.cc:237] T 00000000000000000000000000000000 P df6cb0dbe7cd48d094022931f6b91528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6cb0dbe7cd48d094022931f6b91528" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46731 } }
I20260929 01:06:19.918641 21144 sys_catalog.cc:564] T 00000000000000000000000000000000 P df6cb0dbe7cd48d094022931f6b91528 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:19.919159 21149 sys_catalog.cc:455] T 00000000000000000000000000000000 P df6cb0dbe7cd48d094022931f6b91528 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df6cb0dbe7cd48d094022931f6b91528. Latest consensus state: current_term: 1 leader_uuid: "df6cb0dbe7cd48d094022931f6b91528" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6cb0dbe7cd48d094022931f6b91528" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46731 } } }
I20260929 01:06:19.919275 21149 sys_catalog.cc:458] T 00000000000000000000000000000000 P df6cb0dbe7cd48d094022931f6b91528 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:19.919574 21150 sys_catalog.cc:455] T 00000000000000000000000000000000 P df6cb0dbe7cd48d094022931f6b91528 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df6cb0dbe7cd48d094022931f6b91528" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6cb0dbe7cd48d094022931f6b91528" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46731 } } }
I20260929 01:06:19.919679 21150 sys_catalog.cc:458] T 00000000000000000000000000000000 P df6cb0dbe7cd48d094022931f6b91528 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:19.919694 21157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:19.924548 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:19.919849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46855 (local address 127.0.0.1:46731)
0929 01:06:19.919949 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:19.919953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:19.920123 (+ 170us) server_negotiation.cc:407] Connection header received
0929 01:06:19.920259 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:19.920262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:19.920310 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:19.920402 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:19.920872 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:19.921612 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:19.922815 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:19.923058 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:19.923169 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:19.924055 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:19.924061 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:19.924066 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:19.924079 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:19.924109 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:19.924124 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:19.924189 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:19.924343 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 01:06:19.924396 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 01:06:19.924744 21157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:19.930989 21157 catalog_manager.cc:1347] Generated new cluster ID: 5bd3a048767747259dc78edd00a13d70
I20260929 01:06:19.931061 21157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:19.957046 21157 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:19.958026 21157 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:19.964190 21157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df6cb0dbe7cd48d094022931f6b91528: Generated new TSK 0
I20260929 01:06:19.964963 21157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:19.975907 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 21076
2026-09-29T01:06:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:20 dist-test-slave-7zdl krb5kdc[21175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:20 dist-test-slave-7zdl krb5kdc[21175](info): set up 2 sockets
Sep 29 01:06:20 dist-test-slave-7zdl krb5kdc[21175](info): commencing operation
krb5kdc: starting...
W20260929 01:06:21.023175 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:06:21 dist-test-slave-7zdl krb5kdc[21175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:06:21.143419 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38755
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:21.252640 21188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:21.252890 21188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:21.252936 21188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:21.256820 21188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:21.256876 21188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:21.256896 21188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:21.256911 21188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:21.256932 21188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:21.261976 21188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:21.262953 21188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:21.263895 21188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:21.272027 21196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:21.272034 21194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:21.272065 21193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:21.272258 21188 server_base.cc:1034] running on GCE node
I20260929 01:06:21.272863 21188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:21.273970 21188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:21.275154 21188 hybrid_clock.cc:648] HybridClock initialized: now 1790643981275142 us; error 24 us; skew 500 ppm
Sep 29 01:06:21 dist-test-slave-7zdl krb5kdc[21175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:21.277956 21188 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:06:21.278923 21188 webserver.cc:460] Webserver started at http://127.0.0.1:36511/ using document root <none> and password file <none>
I20260929 01:06:21.279466 21188 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:21.279553 21188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:21.279771 21188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:21.281746 21188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "dd9fa9214c4b4e549bfb7c201b841edb"
format_stamp: "Formatted at 2026-09-29 01:06:21 on dist-test-slave-7zdl"
server_key: "c1bca73d65b7012d3b0483000f2004fc"
server_key_iv: "32cb37767d7d7bf4c9bca7ef0e6d2b9c"
server_key_version: "encryptionkey@0"
I20260929 01:06:21.282307 21188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "dd9fa9214c4b4e549bfb7c201b841edb"
format_stamp: "Formatted at 2026-09-29 01:06:21 on dist-test-slave-7zdl"
server_key: "c1bca73d65b7012d3b0483000f2004fc"
server_key_iv: "32cb37767d7d7bf4c9bca7ef0e6d2b9c"
server_key_version: "encryptionkey@0"
I20260929 01:06:21.285902 21188 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 01:06:21.288275 21203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:21.289438 21188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:06:21.289573 21188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "dd9fa9214c4b4e549bfb7c201b841edb"
format_stamp: "Formatted at 2026-09-29 01:06:21 on dist-test-slave-7zdl"
server_key: "c1bca73d65b7012d3b0483000f2004fc"
server_key_iv: "32cb37767d7d7bf4c9bca7ef0e6d2b9c"
server_key_version: "encryptionkey@0"
I20260929 01:06:21.289691 21188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:21.294155 21188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:21.297838 21188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:21.298110 21188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:21.305470 21188 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38755
I20260929 01:06:21.305509 21255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38755 every 8 connection(s)
I20260929 01:06:21.306435 21188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:21.308759 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 21188
I20260929 01:06:21.308921 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:21.309211 392 external_mini_cluster.cc:1428] Setting key eb968d174f9d2b07112ea92a250a2ed6
I20260929 01:06:21.309643 21256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:21.315011 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:21.310633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33235 (local address 127.0.0.1:38755)
0929 01:06:21.311115 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:21.311135 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:21.311179 (+ 44us) server_negotiation.cc:407] Connection header received
0929 01:06:21.311979 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:21.312003 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:21.312363 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:21.312743 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:21.313459 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:21.313465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:21.313481 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:21.313494 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:21.313556 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:21.313581 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:21.313734 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:21.313850 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 01:06:21.314001 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":180,"threads_started":1}
I20260929 01:06:21.317133 21256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd9fa9214c4b4e549bfb7c201b841edb: Bootstrap starting.
I20260929 01:06:21.318060 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:21.319705 21256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd9fa9214c4b4e549bfb7c201b841edb: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:21.320600 21256 log.cc:824] T 00000000000000000000000000000000 P dd9fa9214c4b4e549bfb7c201b841edb: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:21.321980 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:21.320222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45217 (local address 127.0.0.1:38755)
0929 01:06:21.320354 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:21.320359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:21.320586 (+ 227us) server_negotiation.cc:407] Connection header received
0929 01:06:21.320726 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:21.320730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:21.320775 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:21.320840 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:21.321625 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:21.321629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:21.321631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:21.321643 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:21.321670 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:21.321682 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:21.321775 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:21.321833 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 01:06:21.321868 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:06:21.322975 21256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd9fa9214c4b4e549bfb7c201b841edb: No bootstrap required, opened a new log
I20260929 01:06:21.326531 21256 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd9fa9214c4b4e549bfb7c201b841edb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd9fa9214c4b4e549bfb7c201b841edb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38755 } }
I20260929 01:06:21.326747 21256 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd9fa9214c4b4e549bfb7c201b841edb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:21.326843 21256 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd9fa9214c4b4e549bfb7c201b841edb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd9fa9214c4b4e549bfb7c201b841edb, State: Initialized, Role: FOLLOWER
I20260929 01:06:21.327280 21256 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd9fa9214c4b4e549bfb7c201b841edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd9fa9214c4b4e549bfb7c201b841edb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38755 } }
I20260929 01:06:21.327435 21256 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd9fa9214c4b4e549bfb7c201b841edb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:21.327527 21256 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd9fa9214c4b4e549bfb7c201b841edb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:21.327630 21256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd9fa9214c4b4e549bfb7c201b841edb [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:21.328593 21256 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd9fa9214c4b4e549bfb7c201b841edb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd9fa9214c4b4e549bfb7c201b841edb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38755 } }
I20260929 01:06:21.328925 21256 leader_election.cc:302] T 00000000000000000000000000000000 P dd9fa9214c4b4e549bfb7c201b841edb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd9fa9214c4b4e549bfb7c201b841edb; no voters:
I20260929 01:06:21.329219 21256 leader_election.cc:288] T 00000000000000000000000000000000 P dd9fa9214c4b4e549bfb7c201b841edb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:21.329368 21266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd9fa9214c4b4e549bfb7c201b841edb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:21.329584 21266 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd9fa9214c4b4e549bfb7c201b841edb [term 1 LEADER]: Becoming Leader. State: Replica: dd9fa9214c4b4e549bfb7c201b841edb, State: Running, Role: LEADER
I20260929 01:06:21.329873 21266 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd9fa9214c4b4e549bfb7c201b841edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd9fa9214c4b4e549bfb7c201b841edb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38755 } }
I20260929 01:06:21.330408 21256 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd9fa9214c4b4e549bfb7c201b841edb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:21.331558 21268 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd9fa9214c4b4e549bfb7c201b841edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd9fa9214c4b4e549bfb7c201b841edb. Latest consensus state: current_term: 1 leader_uuid: "dd9fa9214c4b4e549bfb7c201b841edb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd9fa9214c4b4e549bfb7c201b841edb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38755 } } }
I20260929 01:06:21.331694 21268 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd9fa9214c4b4e549bfb7c201b841edb [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:21.332077 21275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:21.332063 21267 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd9fa9214c4b4e549bfb7c201b841edb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd9fa9214c4b4e549bfb7c201b841edb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd9fa9214c4b4e549bfb7c201b841edb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38755 } } }
I20260929 01:06:21.332674 21267 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd9fa9214c4b4e549bfb7c201b841edb [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:21.335180 21275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:21.340440 21275 catalog_manager.cc:1347] Generated new cluster ID: 67a6194ae58f4a9ab35042b3d5b142bf
I20260929 01:06:21.340538 21275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:21.354894 21275 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:21.355240 21275 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:21.360041 21275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd9fa9214c4b4e549bfb7c201b841edb: Generated new TSK 0
I20260929 01:06:21.360651 21275 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:21.371733 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 21188
2026-09-29T01:06:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:21 dist-test-slave-7zdl krb5kdc[21287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:21 dist-test-slave-7zdl krb5kdc[21287](info): set up 2 sockets
Sep 29 01:06:21 dist-test-slave-7zdl krb5kdc[21287](info): commencing operation
krb5kdc: starting...
W20260929 01:06:22.418999 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:06:22 dist-test-slave-7zdl krb5kdc[21287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:06:22.541421 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46201
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:22.649001 21300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:22.649276 21300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:22.649318 21300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:22.653360 21300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:22.653415 21300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:22.653434 21300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:22.653451 21300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:22.653491 21300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:22.658587 21300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:22.659624 21300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:22.660640 21300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:22.668960 21308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:22.669011 21306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:22.669159 21300 server_base.cc:1034] running on GCE node
W20260929 01:06:22.668984 21305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:22.669718 21300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:22.670781 21300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:22.672014 21300 hybrid_clock.cc:648] HybridClock initialized: now 1790643982672000 us; error 54 us; skew 500 ppm
Sep 29 01:06:22 dist-test-slave-7zdl krb5kdc[21287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:22.674686 21300 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:06:22.675522 21300 webserver.cc:460] Webserver started at http://127.0.0.1:44853/ using document root <none> and password file <none>
I20260929 01:06:22.676056 21300 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:22.676146 21300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:22.676388 21300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:22.678418 21300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "1867b38bf45f400a83b9b0e620545c7d"
format_stamp: "Formatted at 2026-09-29 01:06:22 on dist-test-slave-7zdl"
server_key: "e5ee3a0371a07e3e5dd626e3d4b62b45"
server_key_iv: "be12ba50c2cd6935cdab2e3915434ffb"
server_key_version: "encryptionkey@0"
I20260929 01:06:22.679024 21300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "1867b38bf45f400a83b9b0e620545c7d"
format_stamp: "Formatted at 2026-09-29 01:06:22 on dist-test-slave-7zdl"
server_key: "e5ee3a0371a07e3e5dd626e3d4b62b45"
server_key_iv: "be12ba50c2cd6935cdab2e3915434ffb"
server_key_version: "encryptionkey@0"
I20260929 01:06:22.682446 21300 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:06:22.684736 21315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:22.685911 21300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:06:22.686044 21300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "1867b38bf45f400a83b9b0e620545c7d"
format_stamp: "Formatted at 2026-09-29 01:06:22 on dist-test-slave-7zdl"
server_key: "e5ee3a0371a07e3e5dd626e3d4b62b45"
server_key_iv: "be12ba50c2cd6935cdab2e3915434ffb"
server_key_version: "encryptionkey@0"
I20260929 01:06:22.686151 21300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:22.703668 21300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:22.706997 21300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:22.707206 21300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:22.714586 21300 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46201
I20260929 01:06:22.714610 21367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46201 every 8 connection(s)
I20260929 01:06:22.715534 21300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:22.717056 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 21300
I20260929 01:06:22.717200 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:22.717507 392 external_mini_cluster.cc:1428] Setting key cfc410295b8a541477fc0cc9fe9c016f
I20260929 01:06:22.718683 21368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:22.725044 21368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1867b38bf45f400a83b9b0e620545c7d: Bootstrap starting.
Sep 29 01:06:22 dist-test-slave-7zdl krb5kdc[21287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 01:06:22.727636 21368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1867b38bf45f400a83b9b0e620545c7d: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:22.728518 21368 log.cc:824] T 00000000000000000000000000000000 P 1867b38bf45f400a83b9b0e620545c7d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:22.730690 21368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1867b38bf45f400a83b9b0e620545c7d: No bootstrap required, opened a new log
I20260929 01:06:22.732717 21371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:22.719004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:60785 (local address 127.0.0.1:46201)
0929 01:06:22.719389 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:22.719414 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:22.719459 (+ 45us) server_negotiation.cc:407] Connection header received
0929 01:06:22.720121 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:22.720137 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:22.720466 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:22.720755 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:22.721659 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:22.722735 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:22.724010 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:22.724474 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:22.724620 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:22.727594 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:22.727601 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:22.727617 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:22.727648 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:22.729602 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:22.730112 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:22.730118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:22.730123 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:22.730246 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:22.730555 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:22.730558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:22.730559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:22.731083 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:22.731280 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:22.731578 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 01:06:22.731796 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":134,"threads_started":1}
I20260929 01:06:22.734975 21368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1867b38bf45f400a83b9b0e620545c7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1867b38bf45f400a83b9b0e620545c7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46201 } }
I20260929 01:06:22.735226 21368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1867b38bf45f400a83b9b0e620545c7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:22.735280 21368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1867b38bf45f400a83b9b0e620545c7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1867b38bf45f400a83b9b0e620545c7d, State: Initialized, Role: FOLLOWER
I20260929 01:06:22.735725 21368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1867b38bf45f400a83b9b0e620545c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1867b38bf45f400a83b9b0e620545c7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46201 } }
I20260929 01:06:22.735808 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:22.735881 21368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1867b38bf45f400a83b9b0e620545c7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:22.735966 21368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1867b38bf45f400a83b9b0e620545c7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:22.736083 21368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1867b38bf45f400a83b9b0e620545c7d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:22.736953 21368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1867b38bf45f400a83b9b0e620545c7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1867b38bf45f400a83b9b0e620545c7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46201 } }
I20260929 01:06:22.737300 21368 leader_election.cc:302] T 00000000000000000000000000000000 P 1867b38bf45f400a83b9b0e620545c7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1867b38bf45f400a83b9b0e620545c7d; no voters:
I20260929 01:06:22.737614 21368 leader_election.cc:288] T 00000000000000000000000000000000 P 1867b38bf45f400a83b9b0e620545c7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:22.737704 21377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1867b38bf45f400a83b9b0e620545c7d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:22.737946 21377 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1867b38bf45f400a83b9b0e620545c7d [term 1 LEADER]: Becoming Leader. State: Replica: 1867b38bf45f400a83b9b0e620545c7d, State: Running, Role: LEADER
I20260929 01:06:22.738282 21377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1867b38bf45f400a83b9b0e620545c7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1867b38bf45f400a83b9b0e620545c7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46201 } }
I20260929 01:06:22.738916 21368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1867b38bf45f400a83b9b0e620545c7d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:22.740087 21379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1867b38bf45f400a83b9b0e620545c7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1867b38bf45f400a83b9b0e620545c7d. Latest consensus state: current_term: 1 leader_uuid: "1867b38bf45f400a83b9b0e620545c7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1867b38bf45f400a83b9b0e620545c7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46201 } } }
I20260929 01:06:22.740206 21379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1867b38bf45f400a83b9b0e620545c7d [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:22.740517 21378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1867b38bf45f400a83b9b0e620545c7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1867b38bf45f400a83b9b0e620545c7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1867b38bf45f400a83b9b0e620545c7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46201 } } }
I20260929 01:06:22.740579 21387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:22.740635 21378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1867b38bf45f400a83b9b0e620545c7d [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:22.743909 21387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:22.747907 21371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:22.737835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:36939 (local address 127.0.0.1:46201)
0929 01:06:22.737951 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:22.737957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:22.738245 (+ 288us) server_negotiation.cc:407] Connection header received
0929 01:06:22.738337 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:22.738339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:22.738396 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:22.738472 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:22.739406 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:22.740154 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:22.741248 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:22.741621 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:22.741738 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:22.744343 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:22.744347 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:22.744349 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:22.744372 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:22.746163 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:22.746607 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:22.746611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:22.746613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:22.746695 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:22.747044 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:22.747047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:22.747049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:22.747217 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:22.747364 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:22.747637 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 01:06:22.747724 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 01:06:22.750181 21387 catalog_manager.cc:1347] Generated new cluster ID: d7ae61e0f4ae4abe870e9904ae485b45
I20260929 01:06:22.750249 21387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:22.757079 21387 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:22.757908 21387 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:22.767454 21387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1867b38bf45f400a83b9b0e620545c7d: Generated new TSK 0
I20260929 01:06:22.768204 21387 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:22.775722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 21300
W20260929 01:06:22.781149 21285 connection.cc:535] client connection to 127.0.0.1:46201 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46201): Connection reset by peer (error 104)
2026-09-29T01:06:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:22 dist-test-slave-7zdl krb5kdc[21399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:22 dist-test-slave-7zdl krb5kdc[21399](info): set up 2 sockets
Sep 29 01:06:22 dist-test-slave-7zdl krb5kdc[21399](info): commencing operation
krb5kdc: starting...
W20260929 01:06:23.831142 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:06:23 dist-test-slave-7zdl krb5kdc[21399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:06:23.950440 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36689
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:24.057341 21412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:24.057638 21412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:24.057713 21412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:24.061645 21412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:24.061729 21412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:24.061769 21412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:24.061801 21412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:24.061830 21412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:24.066944 21412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:24.067890 21412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:24.068840 21412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:24.076802 21420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:24.076812 21417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:24.076874 21418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:24.076910 21412 server_base.cc:1034] running on GCE node
I20260929 01:06:24.077524 21412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:24.078593 21412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:24.079771 21412 hybrid_clock.cc:648] HybridClock initialized: now 1790643984079736 us; error 46 us; skew 500 ppm
Sep 29 01:06:24 dist-test-slave-7zdl krb5kdc[21399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:24.082372 21412 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:06:24.083274 21412 webserver.cc:460] Webserver started at http://127.0.0.1:35357/ using document root <none> and password file <none>
I20260929 01:06:24.083849 21412 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:24.083936 21412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:24.084167 21412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:24.086174 21412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "98432e8781c94c99953c97df1b7ffd52"
format_stamp: "Formatted at 2026-09-29 01:06:24 on dist-test-slave-7zdl"
server_key: "7a04cf60539853a362ebba6718c1f3ee"
server_key_iv: "d5c743e3e12148b91bc20295a11a0590"
server_key_version: "encryptionkey@0"
I20260929 01:06:24.086714 21412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "98432e8781c94c99953c97df1b7ffd52"
format_stamp: "Formatted at 2026-09-29 01:06:24 on dist-test-slave-7zdl"
server_key: "7a04cf60539853a362ebba6718c1f3ee"
server_key_iv: "d5c743e3e12148b91bc20295a11a0590"
server_key_version: "encryptionkey@0"
I20260929 01:06:24.090123 21412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:06:24.092357 21427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:24.093535 21412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:06:24.093669 21412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "98432e8781c94c99953c97df1b7ffd52"
format_stamp: "Formatted at 2026-09-29 01:06:24 on dist-test-slave-7zdl"
server_key: "7a04cf60539853a362ebba6718c1f3ee"
server_key_iv: "d5c743e3e12148b91bc20295a11a0590"
server_key_version: "encryptionkey@0"
I20260929 01:06:24.093773 21412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:24.109091 21412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:24.111902 21412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:24.112092 21412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:24.118616 21412 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36689
I20260929 01:06:24.118721 21479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36689 every 8 connection(s)
I20260929 01:06:24.119563 21412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:24.122196 21480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:24.125993 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 21412
I20260929 01:06:24.126152 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:24.126400 392 external_mini_cluster.cc:1428] Setting key 502ee54a79b2798948c1904d32ebd9c4
I20260929 01:06:24.127791 21480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98432e8781c94c99953c97df1b7ffd52: Bootstrap starting.
I20260929 01:06:24.130040 21480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98432e8781c94c99953c97df1b7ffd52: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:24.130752 21480 log.cc:824] T 00000000000000000000000000000000 P 98432e8781c94c99953c97df1b7ffd52: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:24.132769 21480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98432e8781c94c99953c97df1b7ffd52: No bootstrap required, opened a new log
I20260929 01:06:24.135164 21483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:24.127792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:47675 (local address 127.0.0.1:36689)
0929 01:06:24.128226 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:24.128245 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:24.128279 (+ 34us) server_negotiation.cc:407] Connection header received
0929 01:06:24.128906 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:24.128922 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:24.129248 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:24.129515 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:24.129854 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:24.130920 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:24.132063 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:24.132507 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:24.132633 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:24.133502 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:24.133506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:24.133518 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:24.133895 (+ 377us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:24.133947 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:24.133968 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:24.134067 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:24.134254 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 01:06:24.134403 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":173,"threads_started":1}
I20260929 01:06:24.136620 21480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98432e8781c94c99953c97df1b7ffd52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98432e8781c94c99953c97df1b7ffd52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36689 } }
I20260929 01:06:24.136873 21480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98432e8781c94c99953c97df1b7ffd52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:24.136956 21480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98432e8781c94c99953c97df1b7ffd52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98432e8781c94c99953c97df1b7ffd52, State: Initialized, Role: FOLLOWER
I20260929 01:06:24.137462 21480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98432e8781c94c99953c97df1b7ffd52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98432e8781c94c99953c97df1b7ffd52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36689 } }
I20260929 01:06:24.137637 21480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98432e8781c94c99953c97df1b7ffd52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:24.137703 21480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98432e8781c94c99953c97df1b7ffd52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:24.137794 21480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98432e8781c94c99953c97df1b7ffd52 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:24.138091 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:24.138731 21480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98432e8781c94c99953c97df1b7ffd52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98432e8781c94c99953c97df1b7ffd52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36689 } }
I20260929 01:06:24.139092 21480 leader_election.cc:302] T 00000000000000000000000000000000 P 98432e8781c94c99953c97df1b7ffd52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98432e8781c94c99953c97df1b7ffd52; no voters:
I20260929 01:06:24.139447 21480 leader_election.cc:288] T 00000000000000000000000000000000 P 98432e8781c94c99953c97df1b7ffd52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:24.139514 21488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98432e8781c94c99953c97df1b7ffd52 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:24.139696 21488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98432e8781c94c99953c97df1b7ffd52 [term 1 LEADER]: Becoming Leader. State: Replica: 98432e8781c94c99953c97df1b7ffd52, State: Running, Role: LEADER
I20260929 01:06:24.140035 21488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98432e8781c94c99953c97df1b7ffd52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98432e8781c94c99953c97df1b7ffd52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36689 } }
I20260929 01:06:24.140667 21480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98432e8781c94c99953c97df1b7ffd52 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:24.141690 21490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98432e8781c94c99953c97df1b7ffd52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98432e8781c94c99953c97df1b7ffd52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98432e8781c94c99953c97df1b7ffd52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36689 } } }
I20260929 01:06:24.141865 21490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98432e8781c94c99953c97df1b7ffd52 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:24.142127 21491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98432e8781c94c99953c97df1b7ffd52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98432e8781c94c99953c97df1b7ffd52. Latest consensus state: current_term: 1 leader_uuid: "98432e8781c94c99953c97df1b7ffd52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98432e8781c94c99953c97df1b7ffd52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36689 } } }
I20260929 01:06:24.142263 21491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98432e8781c94c99953c97df1b7ffd52 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:24.142939 21499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:24.145367 21483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:24.140154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:55011 (local address 127.0.0.1:36689)
0929 01:06:24.140264 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:24.140268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:24.140896 (+ 628us) server_negotiation.cc:407] Connection header received
0929 01:06:24.141071 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:24.141074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:24.141116 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:24.141203 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:24.141658 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:24.142408 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:24.143694 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:24.143904 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:24.143986 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:24.144888 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:24.144891 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:24.144894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:24.144906 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:24.144933 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:24.144947 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:24.145036 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:24.145201 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 01:06:24.145233 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 01:06:24.146106 21499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:24.151113 21499 catalog_manager.cc:1347] Generated new cluster ID: 41015bdb3d2947a0a4be30eeda712b2b
I20260929 01:06:24.151177 21499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:24.163797 21499 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:24.165019 21499 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:24.178604 21499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98432e8781c94c99953c97df1b7ffd52: Generated new TSK 0
I20260929 01:06:24.179453 21499 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:24.188167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 21412
W20260929 01:06:24.193745 21397 connection.cc:535] client connection to 127.0.0.1:36689 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36689): Connection reset by peer (error 104)
2026-09-29T01:06:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:24 dist-test-slave-7zdl krb5kdc[21511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:24 dist-test-slave-7zdl krb5kdc[21511](info): set up 2 sockets
Sep 29 01:06:24 dist-test-slave-7zdl krb5kdc[21511](info): commencing operation
krb5kdc: starting...
W20260929 01:06:25.227018 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:06:25 dist-test-slave-7zdl krb5kdc[21511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:06:25.346036 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43579
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:25.454399 21524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:25.454658 21524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:25.454698 21524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:25.458642 21524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:25.458700 21524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:25.458719 21524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:25.458737 21524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:25.458752 21524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:25.463991 21524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:25.464923 21524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:25.465842 21524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:25.474092 21529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:25.474107 21532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:25.474263 21524 server_base.cc:1034] running on GCE node
W20260929 01:06:25.474181 21530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:25.474924 21524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:25.476016 21524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:25.477174 21524 hybrid_clock.cc:648] HybridClock initialized: now 1790643985477148 us; error 38 us; skew 500 ppm
Sep 29 01:06:25 dist-test-slave-7zdl krb5kdc[21511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:25.487025 21524 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:06:25.487949 21524 webserver.cc:460] Webserver started at http://127.0.0.1:42121/ using document root <none> and password file <none>
I20260929 01:06:25.488505 21524 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:25.488592 21524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:25.488817 21524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:25.490867 21524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "58c329ce015440178452d0b211dff722"
format_stamp: "Formatted at 2026-09-29 01:06:25 on dist-test-slave-7zdl"
server_key: "f72524fa6bddf25b29f3ee9dbb180ad5"
server_key_iv: "470185813677fffab67e2b86905c9798"
server_key_version: "encryptionkey@0"
I20260929 01:06:25.491454 21524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "58c329ce015440178452d0b211dff722"
format_stamp: "Formatted at 2026-09-29 01:06:25 on dist-test-slave-7zdl"
server_key: "f72524fa6bddf25b29f3ee9dbb180ad5"
server_key_iv: "470185813677fffab67e2b86905c9798"
server_key_version: "encryptionkey@0"
I20260929 01:06:25.494971 21524 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:06:25.497391 21539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:25.498737 21524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:06:25.498914 21524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "58c329ce015440178452d0b211dff722"
format_stamp: "Formatted at 2026-09-29 01:06:25 on dist-test-slave-7zdl"
server_key: "f72524fa6bddf25b29f3ee9dbb180ad5"
server_key_iv: "470185813677fffab67e2b86905c9798"
server_key_version: "encryptionkey@0"
I20260929 01:06:25.499066 21524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:25.503605 21524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:25.506708 21524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:25.506916 21524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:25.513983 21524 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43579
I20260929 01:06:25.513999 21591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43579 every 8 connection(s)
I20260929 01:06:25.514961 21524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:25.517522 21592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:25.521643 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 21524
I20260929 01:06:25.521806 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:25.522122 392 external_mini_cluster.cc:1428] Setting key dd0f0ed041f7d87103d9c4b7913220ff
I20260929 01:06:25.523844 21592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58c329ce015440178452d0b211dff722: Bootstrap starting.
I20260929 01:06:25.526052 21592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58c329ce015440178452d0b211dff722: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:25.526790 21592 log.cc:824] T 00000000000000000000000000000000 P 58c329ce015440178452d0b211dff722: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:25.527786 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:25.523841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:58215 (local address 127.0.0.1:43579)
0929 01:06:25.524284 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:25.524310 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:25.524350 (+ 40us) server_negotiation.cc:407] Connection header received
0929 01:06:25.524995 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:25.525010 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:25.525284 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:25.525615 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:25.526423 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:25.526428 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:25.526449 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:25.526460 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:25.526511 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:25.526530 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:25.526643 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:25.526745 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 01:06:25.526922 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":179,"threads_started":1}
I20260929 01:06:25.528777 21592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58c329ce015440178452d0b211dff722: No bootstrap required, opened a new log
I20260929 01:06:25.530578 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:25.533215 21592 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58c329ce015440178452d0b211dff722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c329ce015440178452d0b211dff722" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43579 } }
I20260929 01:06:25.533484 21592 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58c329ce015440178452d0b211dff722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:25.533588 21592 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58c329ce015440178452d0b211dff722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58c329ce015440178452d0b211dff722, State: Initialized, Role: FOLLOWER
I20260929 01:06:25.534080 21592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58c329ce015440178452d0b211dff722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c329ce015440178452d0b211dff722" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43579 } }
I20260929 01:06:25.534252 21592 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58c329ce015440178452d0b211dff722 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:25.534322 21592 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58c329ce015440178452d0b211dff722 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:25.534413 21592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58c329ce015440178452d0b211dff722 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:25.534869 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:25.532991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:45763 (local address 127.0.0.1:43579)
0929 01:06:25.533147 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:25.533152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:25.533399 (+ 247us) server_negotiation.cc:407] Connection header received
0929 01:06:25.533571 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:25.533575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:25.533640 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:25.533772 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:25.534479 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:25.534482 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:25.534485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:25.534499 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:25.534526 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:25.534539 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:25.534626 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:25.534701 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 01:06:25.534740 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 01:06:25.535560 21592 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58c329ce015440178452d0b211dff722 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c329ce015440178452d0b211dff722" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43579 } }
I20260929 01:06:25.535892 21592 leader_election.cc:302] T 00000000000000000000000000000000 P 58c329ce015440178452d0b211dff722 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58c329ce015440178452d0b211dff722; no voters:
I20260929 01:06:25.536180 21592 leader_election.cc:288] T 00000000000000000000000000000000 P 58c329ce015440178452d0b211dff722 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:25.536276 21602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58c329ce015440178452d0b211dff722 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:25.536492 21602 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58c329ce015440178452d0b211dff722 [term 1 LEADER]: Becoming Leader. State: Replica: 58c329ce015440178452d0b211dff722, State: Running, Role: LEADER
I20260929 01:06:25.536837 21602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58c329ce015440178452d0b211dff722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c329ce015440178452d0b211dff722" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43579 } }
I20260929 01:06:25.537407 21592 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58c329ce015440178452d0b211dff722 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:25.538767 21604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58c329ce015440178452d0b211dff722 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58c329ce015440178452d0b211dff722. Latest consensus state: current_term: 1 leader_uuid: "58c329ce015440178452d0b211dff722" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c329ce015440178452d0b211dff722" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43579 } } }
I20260929 01:06:25.538911 21604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58c329ce015440178452d0b211dff722 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:25.538867 21603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58c329ce015440178452d0b211dff722 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58c329ce015440178452d0b211dff722" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c329ce015440178452d0b211dff722" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43579 } } }
I20260929 01:06:25.538973 21603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58c329ce015440178452d0b211dff722 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:25.539386 21611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:25.542089 21611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:25.547068 21611 catalog_manager.cc:1347] Generated new cluster ID: 7cdf04dda3694b57b7128f8b2a8dddd6
I20260929 01:06:25.547132 21611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:25.568655 21611 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:25.568969 21611 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:25.578658 21611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58c329ce015440178452d0b211dff722: Generated new TSK 0
I20260929 01:06:25.579336 21611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:25.583103 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 21524
2026-09-29T01:06:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1394 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:25 dist-test-slave-7zdl krb5kdc[21623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:25 dist-test-slave-7zdl krb5kdc[21623](info): set up 2 sockets
Sep 29 01:06:25 dist-test-slave-7zdl krb5kdc[21623](info): commencing operation
krb5kdc: starting...
W20260929 01:06:26.615083 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:06:26 dist-test-slave-7zdl krb5kdc[21623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:06:26.738268 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43097
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:26.846261 21636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:26.846653 21636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:26.846771 21636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:26.850907 21636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:26.851011 21636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:26.851052 21636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:26.851106 21636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:26.851146 21636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:26.856325 21636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43097
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:26.857376 21636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:26.858466 21636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:26.866770 21641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:26.866757 21642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:26.866753 21644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:26.867139 21636 server_base.cc:1034] running on GCE node
I20260929 01:06:26.867647 21636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:26.868773 21636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:26.870000 21636 hybrid_clock.cc:648] HybridClock initialized: now 1790643986869988 us; error 26 us; skew 500 ppm
Sep 29 01:06:26 dist-test-slave-7zdl krb5kdc[21623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:26.872794 21636 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:06:26.873793 21636 webserver.cc:460] Webserver started at http://127.0.0.1:33611/ using document root <none> and password file <none>
I20260929 01:06:26.874464 21636 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:26.874562 21636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:26.874850 21636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:26.877050 21636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "0918dc23d6e14c7b9c4276fff73b3940"
format_stamp: "Formatted at 2026-09-29 01:06:26 on dist-test-slave-7zdl"
server_key: "bc7b6c399a1675ab36e7d1c95560f600"
server_key_iv: "26d3a8e207ae62fee165c2c9bb3f4429"
server_key_version: "encryptionkey@0"
I20260929 01:06:26.877660 21636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "0918dc23d6e14c7b9c4276fff73b3940"
format_stamp: "Formatted at 2026-09-29 01:06:26 on dist-test-slave-7zdl"
server_key: "bc7b6c399a1675ab36e7d1c95560f600"
server_key_iv: "26d3a8e207ae62fee165c2c9bb3f4429"
server_key_version: "encryptionkey@0"
I20260929 01:06:26.881563 21636 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 01:06:26.884254 21651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:26.885540 21636 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:06:26.885704 21636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "0918dc23d6e14c7b9c4276fff73b3940"
format_stamp: "Formatted at 2026-09-29 01:06:26 on dist-test-slave-7zdl"
server_key: "bc7b6c399a1675ab36e7d1c95560f600"
server_key_iv: "26d3a8e207ae62fee165c2c9bb3f4429"
server_key_version: "encryptionkey@0"
I20260929 01:06:26.885838 21636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:26.895553 21636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:26.898770 21636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:26.899062 21636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:26.906397 21703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43097 every 8 connection(s)
I20260929 01:06:26.906395 21636 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43097
I20260929 01:06:26.907487 21636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:26.910202 21704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:26.914042 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 21636
I20260929 01:06:26.914222 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:26.914532 392 external_mini_cluster.cc:1428] Setting key 96514613b03c5f811ccdfbe37f4adc2a
I20260929 01:06:26.915812 21704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0918dc23d6e14c7b9c4276fff73b3940: Bootstrap starting.
I20260929 01:06:26.918411 21704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0918dc23d6e14c7b9c4276fff73b3940: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:26.919284 21704 log.cc:824] T 00000000000000000000000000000000 P 0918dc23d6e14c7b9c4276fff73b3940: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:26.921272 21704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0918dc23d6e14c7b9c4276fff73b3940: No bootstrap required, opened a new log
Sep 29 01:06:26 dist-test-slave-7zdl krb5kdc[21623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 01:06:26.925134 21704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0918dc23d6e14c7b9c4276fff73b3940 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0918dc23d6e14c7b9c4276fff73b3940" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43097 } }
I20260929 01:06:26.925422 21704 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0918dc23d6e14c7b9c4276fff73b3940 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:26.925483 21704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0918dc23d6e14c7b9c4276fff73b3940 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0918dc23d6e14c7b9c4276fff73b3940, State: Initialized, Role: FOLLOWER
I20260929 01:06:26.926016 21704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0918dc23d6e14c7b9c4276fff73b3940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0918dc23d6e14c7b9c4276fff73b3940" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43097 } }
I20260929 01:06:26.926174 21704 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0918dc23d6e14c7b9c4276fff73b3940 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:26.926239 21704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0918dc23d6e14c7b9c4276fff73b3940 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:26.926334 21704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0918dc23d6e14c7b9c4276fff73b3940 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:26.927451 21704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0918dc23d6e14c7b9c4276fff73b3940 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0918dc23d6e14c7b9c4276fff73b3940" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43097 } }
I20260929 01:06:26.927814 21704 leader_election.cc:302] T 00000000000000000000000000000000 P 0918dc23d6e14c7b9c4276fff73b3940 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0918dc23d6e14c7b9c4276fff73b3940; no voters:
I20260929 01:06:26.928144 21704 leader_election.cc:288] T 00000000000000000000000000000000 P 0918dc23d6e14c7b9c4276fff73b3940 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:26.928205 21709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0918dc23d6e14c7b9c4276fff73b3940 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:26.928406 21709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0918dc23d6e14c7b9c4276fff73b3940 [term 1 LEADER]: Becoming Leader. State: Replica: 0918dc23d6e14c7b9c4276fff73b3940, State: Running, Role: LEADER
I20260929 01:06:26.928735 21709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0918dc23d6e14c7b9c4276fff73b3940 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0918dc23d6e14c7b9c4276fff73b3940" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43097 } }
I20260929 01:06:26.929415 21704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0918dc23d6e14c7b9c4276fff73b3940 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:26.929962 21710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0918dc23d6e14c7b9c4276fff73b3940 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0918dc23d6e14c7b9c4276fff73b3940" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0918dc23d6e14c7b9c4276fff73b3940" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43097 } } }
I20260929 01:06:26.930063 21707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:26.916123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:34853 (local address 127.0.0.1:43097)
0929 01:06:26.916525 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:26.916543 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:26.916584 (+ 41us) server_negotiation.cc:407] Connection header received
0929 01:06:26.917430 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:26.917453 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:26.917865 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:26.918222 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:26.919050 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:26.920383 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:26.921236 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:26.921898 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:26.922063 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:26.924575 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:26.924582 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:26.924600 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:26.924630 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:26.926672 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:26.927194 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:26.927200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:26.927206 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:26.927355 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:26.927731 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:26.927733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:26.927735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:26.928212 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:26.928532 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:26.928839 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 01:06:26.929079 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":110,"threads_started":1}
I20260929 01:06:26.930159 21710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0918dc23d6e14c7b9c4276fff73b3940 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:26.930555 21710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0918dc23d6e14c7b9c4276fff73b3940 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0918dc23d6e14c7b9c4276fff73b3940. Latest consensus state: current_term: 1 leader_uuid: "0918dc23d6e14c7b9c4276fff73b3940" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0918dc23d6e14c7b9c4276fff73b3940" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43097 } } }
I20260929 01:06:26.930702 21710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0918dc23d6e14c7b9c4276fff73b3940 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:26.931047 21718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:26.933233 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:26.934670 21718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:26.940663 21718 catalog_manager.cc:1347] Generated new cluster ID: f92f388fccff4faab03aa9da77c1581e
I20260929 01:06:26.940719 21718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:26.944211 21707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:26.935363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:55369 (local address 127.0.0.1:43097)
0929 01:06:26.935455 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:26.935459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:26.935937 (+ 478us) server_negotiation.cc:407] Connection header received
0929 01:06:26.936052 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:26.936055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:26.936097 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:26.936186 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:26.936923 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:26.937751 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:26.938817 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:26.939091 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:26.939201 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:26.940722 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:26.940729 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:26.940733 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:26.940758 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:26.942384 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:26.942966 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:26.942969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:26.942971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:26.943045 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:26.943385 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:26.943388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:26.943390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:26.943567 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:26.943695 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:26.943986 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 01:06:26.944070 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 01:06:26.951370 21718 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:26.952224 21718 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:26.962885 21718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0918dc23d6e14c7b9c4276fff73b3940: Generated new TSK 0
I20260929 01:06:26.963694 21718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:26.971036 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 21636
W20260929 01:06:26.976171 21621 connection.cc:535] client connection to 127.0.0.1:43097 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43097): Connection reset by peer (error 104)
2026-09-29T01:06:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:27 dist-test-slave-7zdl krb5kdc[21735](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:27 dist-test-slave-7zdl krb5kdc[21735](info): set up 2 sockets
Sep 29 01:06:27 dist-test-slave-7zdl krb5kdc[21735](info): commencing operation
krb5kdc: starting...
W20260929 01:06:28.015177 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:06:28 dist-test-slave-7zdl krb5kdc[21735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:06:28.133930 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35285
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:28.243958 21748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:28.244210 21748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:28.244278 21748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:28.248344 21748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:28.248401 21748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:28.248423 21748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:28.248462 21748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:28.248481 21748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:28.253454 21748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35285
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:28.254441 21748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:28.255486 21748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:28.263691 21753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:28.263948 21754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:28.263993 21756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:28.263949 21748 server_base.cc:1034] running on GCE node
I20260929 01:06:28.264570 21748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:28.265650 21748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:28.266894 21748 hybrid_clock.cc:648] HybridClock initialized: now 1790643988266865 us; error 30 us; skew 500 ppm
Sep 29 01:06:28 dist-test-slave-7zdl krb5kdc[21735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:28.269528 21748 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:06:28.270437 21748 webserver.cc:460] Webserver started at http://127.0.0.1:42143/ using document root <none> and password file <none>
I20260929 01:06:28.271090 21748 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:28.271190 21748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:28.271432 21748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:28.273388 21748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "1e63e8baf33a49cda14e32784f8aa147"
format_stamp: "Formatted at 2026-09-29 01:06:28 on dist-test-slave-7zdl"
server_key: "14d8e9333d1be7c990505f5191cf4d8b"
server_key_iv: "b0654bfc446d2231e4f2fbaac24d7c1c"
server_key_version: "encryptionkey@0"
I20260929 01:06:28.273999 21748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "1e63e8baf33a49cda14e32784f8aa147"
format_stamp: "Formatted at 2026-09-29 01:06:28 on dist-test-slave-7zdl"
server_key: "14d8e9333d1be7c990505f5191cf4d8b"
server_key_iv: "b0654bfc446d2231e4f2fbaac24d7c1c"
server_key_version: "encryptionkey@0"
I20260929 01:06:28.277736 21748 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 01:06:28.280272 21763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:28.281414 21748 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:06:28.281572 21748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "1e63e8baf33a49cda14e32784f8aa147"
format_stamp: "Formatted at 2026-09-29 01:06:28 on dist-test-slave-7zdl"
server_key: "14d8e9333d1be7c990505f5191cf4d8b"
server_key_iv: "b0654bfc446d2231e4f2fbaac24d7c1c"
server_key_version: "encryptionkey@0"
I20260929 01:06:28.281687 21748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:28.295872 21748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:28.298568 21748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:28.298779 21748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:28.306103 21748 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35285
I20260929 01:06:28.306110 21815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35285 every 8 connection(s)
I20260929 01:06:28.307052 21748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:28.309579 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 21748
I20260929 01:06:28.309717 21816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:28.309741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:28.310113 392 external_mini_cluster.cc:1428] Setting key 3ef2c3191731cde3ba7a757bbbe567a1
I20260929 01:06:28.316272 21816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e63e8baf33a49cda14e32784f8aa147: Bootstrap starting.
I20260929 01:06:28.318748 21816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e63e8baf33a49cda14e32784f8aa147: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:28.319278 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:28.311440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:38101 (local address 127.0.0.1:35285)
0929 01:06:28.311873 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:28.311901 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:28.311938 (+ 37us) server_negotiation.cc:407] Connection header received
0929 01:06:28.312615 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:28.312632 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:28.312958 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:28.313227 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:28.313698 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:28.314732 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:28.315969 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:28.316472 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:28.316594 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:28.317471 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:28.317475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:28.317487 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:28.317902 (+ 415us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:28.317956 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:28.317978 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:28.318077 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:28.318269 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 01:06:28.318420 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":164,"threads_started":1}
I20260929 01:06:28.319542 21816 log.cc:824] T 00000000000000000000000000000000 P 1e63e8baf33a49cda14e32784f8aa147: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:28.321534 21816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e63e8baf33a49cda14e32784f8aa147: No bootstrap required, opened a new log
I20260929 01:06:28.322077 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:28.325589 21816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e63e8baf33a49cda14e32784f8aa147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e63e8baf33a49cda14e32784f8aa147" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35285 } }
I20260929 01:06:28.325812 21816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e63e8baf33a49cda14e32784f8aa147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:28.325878 21816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e63e8baf33a49cda14e32784f8aa147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e63e8baf33a49cda14e32784f8aa147, State: Initialized, Role: FOLLOWER
I20260929 01:06:28.326370 21816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e63e8baf33a49cda14e32784f8aa147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e63e8baf33a49cda14e32784f8aa147" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35285 } }
I20260929 01:06:28.326524 21816 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e63e8baf33a49cda14e32784f8aa147 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:28.326583 21816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e63e8baf33a49cda14e32784f8aa147 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:28.326687 21816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e63e8baf33a49cda14e32784f8aa147 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:28.328117 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:28.324046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:60691 (local address 127.0.0.1:35285)
0929 01:06:28.324174 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:28.324179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:28.324268 (+ 89us) server_negotiation.cc:407] Connection header received
0929 01:06:28.324341 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:28.324345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:28.324400 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:28.324488 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:28.324705 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:28.325452 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:28.326314 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:28.326671 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:28.326784 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:28.327656 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:28.327660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:28.327663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:28.327675 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:28.327705 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:28.327723 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:28.327820 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:28.327929 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 01:06:28.327965 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 01:06:28.328081 21816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e63e8baf33a49cda14e32784f8aa147 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e63e8baf33a49cda14e32784f8aa147" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35285 } }
I20260929 01:06:28.328450 21816 leader_election.cc:302] T 00000000000000000000000000000000 P 1e63e8baf33a49cda14e32784f8aa147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e63e8baf33a49cda14e32784f8aa147; no voters:
I20260929 01:06:28.328779 21816 leader_election.cc:288] T 00000000000000000000000000000000 P 1e63e8baf33a49cda14e32784f8aa147 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:28.328871 21826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e63e8baf33a49cda14e32784f8aa147 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:28.329104 21826 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e63e8baf33a49cda14e32784f8aa147 [term 1 LEADER]: Becoming Leader. State: Replica: 1e63e8baf33a49cda14e32784f8aa147, State: Running, Role: LEADER
I20260929 01:06:28.329381 21826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e63e8baf33a49cda14e32784f8aa147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e63e8baf33a49cda14e32784f8aa147" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35285 } }
I20260929 01:06:28.329973 21816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e63e8baf33a49cda14e32784f8aa147 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:28.330442 21826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e63e8baf33a49cda14e32784f8aa147 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e63e8baf33a49cda14e32784f8aa147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e63e8baf33a49cda14e32784f8aa147" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35285 } } }
I20260929 01:06:28.330631 21826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e63e8baf33a49cda14e32784f8aa147 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:28.330878 21826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e63e8baf33a49cda14e32784f8aa147 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e63e8baf33a49cda14e32784f8aa147. Latest consensus state: current_term: 1 leader_uuid: "1e63e8baf33a49cda14e32784f8aa147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e63e8baf33a49cda14e32784f8aa147" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35285 } } }
I20260929 01:06:28.330976 21826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e63e8baf33a49cda14e32784f8aa147 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:28.331104 21834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:28.334156 21834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:28.339704 21834 catalog_manager.cc:1347] Generated new cluster ID: 1d5dd345a5e14d058cc48b6948354c16
I20260929 01:06:28.339768 21834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:28.363808 21834 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:28.365056 21834 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:28.382627 21834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e63e8baf33a49cda14e32784f8aa147: Generated new TSK 0
I20260929 01:06:28.383459 21834 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:28.396075 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 21748
W20260929 01:06:28.400616 21733 connection.cc:535] client connection to 127.0.0.1:35285 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35285): Connection reset by peer (error 104)
2026-09-29T01:06:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:28 dist-test-slave-7zdl krb5kdc[21847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:28 dist-test-slave-7zdl krb5kdc[21847](info): set up 2 sockets
Sep 29 01:06:28 dist-test-slave-7zdl krb5kdc[21847](info): commencing operation
krb5kdc: starting...
W20260929 01:06:29.430898 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:06:29 dist-test-slave-7zdl krb5kdc[21847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 01:06:29.557788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35589
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:29.664268 21860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:29.664546 21860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:29.664621 21860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:29.668742 21860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:29.668869 21860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:29.668916 21860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:29.668953 21860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:29.668984 21860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:29.674552 21860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35589
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:29.675668 21860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:29.676698 21860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:06:29.685408 21860 server_base.cc:1034] running on GCE node
W20260929 01:06:29.685360 21865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:29.685400 21866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:29.685360 21868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:29.686193 21860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:29.687269 21860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:29.688460 21860 hybrid_clock.cc:648] HybridClock initialized: now 1790643989688437 us; error 35 us; skew 500 ppm
Sep 29 01:06:29 dist-test-slave-7zdl krb5kdc[21847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:29.691179 21860 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 01:06:29.692016 21860 webserver.cc:460] Webserver started at http://127.0.0.1:39867/ using document root <none> and password file <none>
I20260929 01:06:29.692499 21860 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:29.692564 21860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:29.692780 21860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:29.694718 21860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "87c3cd14bf844bc498a23b978c092db3"
format_stamp: "Formatted at 2026-09-29 01:06:29 on dist-test-slave-7zdl"
server_key: "f8182631a03508c2ebb521d32607e084"
server_key_iv: "a8d5dd18ff9354ff386f8b136225396b"
server_key_version: "encryptionkey@0"
I20260929 01:06:29.695326 21860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "87c3cd14bf844bc498a23b978c092db3"
format_stamp: "Formatted at 2026-09-29 01:06:29 on dist-test-slave-7zdl"
server_key: "f8182631a03508c2ebb521d32607e084"
server_key_iv: "a8d5dd18ff9354ff386f8b136225396b"
server_key_version: "encryptionkey@0"
I20260929 01:06:29.698982 21860 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:06:29.701277 21875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:29.702425 21860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:06:29.702574 21860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "87c3cd14bf844bc498a23b978c092db3"
format_stamp: "Formatted at 2026-09-29 01:06:29 on dist-test-slave-7zdl"
server_key: "f8182631a03508c2ebb521d32607e084"
server_key_iv: "a8d5dd18ff9354ff386f8b136225396b"
server_key_version: "encryptionkey@0"
I20260929 01:06:29.702672 21860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:29.707767 21860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:29.711400 21860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:29.711575 21860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:29.718183 21860 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35589
I20260929 01:06:29.718230 21927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35589 every 8 connection(s)
I20260929 01:06:29.719195 21860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:29.721971 21928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:29.723217 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 21860
I20260929 01:06:29.723362 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:29.723641 392 external_mini_cluster.cc:1428] Setting key d2320c1b8a1f22e8c19f0bf90c2dcaae
I20260929 01:06:29.728549 21928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c3cd14bf844bc498a23b978c092db3: Bootstrap starting.
I20260929 01:06:29.729398 21931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:29.725174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:33635 (local address 127.0.0.1:35589)
0929 01:06:29.725603 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:29.725622 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:29.725669 (+ 47us) server_negotiation.cc:407] Connection header received
0929 01:06:29.726454 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:29.726479 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:29.726881 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:29.727235 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:29.728050 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:29.728056 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:29.728070 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:29.728082 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:29.728144 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:29.728169 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:29.728306 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:29.728370 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 01:06:29.728518 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":125,"threads_started":1}
I20260929 01:06:29.730924 21928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87c3cd14bf844bc498a23b978c092db3: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:29.731770 21928 log.cc:824] T 00000000000000000000000000000000 P 87c3cd14bf844bc498a23b978c092db3: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:29.732194 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:29.733804 21928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c3cd14bf844bc498a23b978c092db3: No bootstrap required, opened a new log
I20260929 01:06:29.736204 21931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:29.734503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:37099 (local address 127.0.0.1:35589)
0929 01:06:29.734637 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:29.734640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:29.734888 (+ 248us) server_negotiation.cc:407] Connection header received
0929 01:06:29.734983 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:29.734985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:29.735015 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:29.735090 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:29.735779 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:29.735781 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:29.735784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:29.735795 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:29.735870 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:29.735883 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:29.735962 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:29.736079 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 01:06:29.736114 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:06:29.737726 21928 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87c3cd14bf844bc498a23b978c092db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c3cd14bf844bc498a23b978c092db3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35589 } }
I20260929 01:06:29.737929 21928 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87c3cd14bf844bc498a23b978c092db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:29.737998 21928 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87c3cd14bf844bc498a23b978c092db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c3cd14bf844bc498a23b978c092db3, State: Initialized, Role: FOLLOWER
I20260929 01:06:29.738449 21928 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87c3cd14bf844bc498a23b978c092db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c3cd14bf844bc498a23b978c092db3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35589 } }
I20260929 01:06:29.738574 21928 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87c3cd14bf844bc498a23b978c092db3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:29.738615 21928 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87c3cd14bf844bc498a23b978c092db3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:29.738734 21928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87c3cd14bf844bc498a23b978c092db3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:29.739607 21928 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87c3cd14bf844bc498a23b978c092db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c3cd14bf844bc498a23b978c092db3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35589 } }
I20260929 01:06:29.739936 21928 leader_election.cc:302] T 00000000000000000000000000000000 P 87c3cd14bf844bc498a23b978c092db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87c3cd14bf844bc498a23b978c092db3; no voters:
I20260929 01:06:29.740278 21928 leader_election.cc:288] T 00000000000000000000000000000000 P 87c3cd14bf844bc498a23b978c092db3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:29.740343 21938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87c3cd14bf844bc498a23b978c092db3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:29.740590 21938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87c3cd14bf844bc498a23b978c092db3 [term 1 LEADER]: Becoming Leader. State: Replica: 87c3cd14bf844bc498a23b978c092db3, State: Running, Role: LEADER
I20260929 01:06:29.740900 21938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87c3cd14bf844bc498a23b978c092db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c3cd14bf844bc498a23b978c092db3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35589 } }
I20260929 01:06:29.741577 21928 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87c3cd14bf844bc498a23b978c092db3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:29.742692 21939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c3cd14bf844bc498a23b978c092db3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87c3cd14bf844bc498a23b978c092db3. Latest consensus state: current_term: 1 leader_uuid: "87c3cd14bf844bc498a23b978c092db3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c3cd14bf844bc498a23b978c092db3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35589 } } }
I20260929 01:06:29.742727 21940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c3cd14bf844bc498a23b978c092db3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87c3cd14bf844bc498a23b978c092db3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c3cd14bf844bc498a23b978c092db3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35589 } } }
I20260929 01:06:29.742885 21939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c3cd14bf844bc498a23b978c092db3 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:29.742887 21940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c3cd14bf844bc498a23b978c092db3 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:29.743350 21947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:29.746497 21947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:29.752022 21947 catalog_manager.cc:1347] Generated new cluster ID: ae39563efa3c453a9fd39b837c4bf4ff
I20260929 01:06:29.752135 21947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:29.790246 21947 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:29.790617 21947 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:29.799897 21947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87c3cd14bf844bc498a23b978c092db3: Generated new TSK 0
I20260929 01:06:29.800768 21947 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:29.816073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 21860
2026-09-29T01:06:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:29 dist-test-slave-7zdl krb5kdc[21959](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:29 dist-test-slave-7zdl krb5kdc[21959](info): set up 2 sockets
Sep 29 01:06:29 dist-test-slave-7zdl krb5kdc[21959](info): commencing operation
krb5kdc: starting...
W20260929 01:06:30.859036 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:06:30 dist-test-slave-7zdl krb5kdc[21959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:06:30.981544 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38531
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:31.087765 21972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:31.088040 21972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:31.088126 21972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:31.092082 21972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:31.092159 21972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:31.092195 21972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:31.092243 21972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:31.092279 21972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:31.097409 21972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38531
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:31.098385 21972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:31.099381 21972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:31.107626 21978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:31.107772 21972 server_base.cc:1034] running on GCE node
W20260929 01:06:31.107613 21980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:31.107605 21977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:31.108367 21972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:31.109370 21972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:31.110563 21972 hybrid_clock.cc:648] HybridClock initialized: now 1790643991110536 us; error 46 us; skew 500 ppm
Sep 29 01:06:31 dist-test-slave-7zdl krb5kdc[21959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:31.116675 21972 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:06:31.117595 21972 webserver.cc:460] Webserver started at http://127.0.98.62:33987/ using document root <none> and password file <none>
I20260929 01:06:31.118167 21972 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:31.118261 21972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:31.118506 21972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:31.120761 21972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "8a8d1015f699403f994836da202b9741"
format_stamp: "Formatted at 2026-09-29 01:06:31 on dist-test-slave-7zdl"
server_key: "b5989bd1d7b1214411cfbf1835e31184"
server_key_iv: "dfdacabcf96f86d0e82e30b9bc94cc41"
server_key_version: "encryptionkey@0"
I20260929 01:06:31.121457 21972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "8a8d1015f699403f994836da202b9741"
format_stamp: "Formatted at 2026-09-29 01:06:31 on dist-test-slave-7zdl"
server_key: "b5989bd1d7b1214411cfbf1835e31184"
server_key_iv: "dfdacabcf96f86d0e82e30b9bc94cc41"
server_key_version: "encryptionkey@0"
I20260929 01:06:31.125401 21972 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 01:06:31.128172 21987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:31.129385 21972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:06:31.129555 21972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "8a8d1015f699403f994836da202b9741"
format_stamp: "Formatted at 2026-09-29 01:06:31 on dist-test-slave-7zdl"
server_key: "b5989bd1d7b1214411cfbf1835e31184"
server_key_iv: "dfdacabcf96f86d0e82e30b9bc94cc41"
server_key_version: "encryptionkey@0"
I20260929 01:06:31.129667 21972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:31.144673 21972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:31.147826 21972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:31.148036 21972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:31.155537 21972 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38531
I20260929 01:06:31.155556 22039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38531 every 8 connection(s)
I20260929 01:06:31.156507 21972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:31.156963 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 21972
I20260929 01:06:31.157155 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:31.157478 392 external_mini_cluster.cc:1428] Setting key 9fb2b1fbfd9b0b6e3be595321fc93bae
I20260929 01:06:31.160254 22040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 01:06:31 dist-test-slave-7zdl krb5kdc[21959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:06:31.165953 22040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a8d1015f699403f994836da202b9741: Bootstrap starting.
I20260929 01:06:31.168322 22040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a8d1015f699403f994836da202b9741: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:31.169054 22040 log.cc:824] T 00000000000000000000000000000000 P 8a8d1015f699403f994836da202b9741: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:31.170981 22040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a8d1015f699403f994836da202b9741: No bootstrap required, opened a new log
I20260929 01:06:31.171126 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:31.158748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51513 (local address 127.0.98.62:38531)
0929 01:06:31.159213 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:31.159228 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:31.159261 (+ 33us) server_negotiation.cc:407] Connection header received
0929 01:06:31.159874 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:31.159889 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:31.160225 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:31.160616 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:31.161409 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:31.162376 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:31.163299 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:31.163775 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:31.163928 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:31.166592 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:31.166596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:31.166608 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:31.166633 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:31.168251 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:31.168760 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:31.168765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:31.168769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:31.168870 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:31.169203 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:31.169206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:31.169208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:31.169591 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:31.169800 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:31.170128 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 01:06:31.170337 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":138,"threads_started":1}
I20260929 01:06:31.173974 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:31.175097 22040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a8d1015f699403f994836da202b9741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8d1015f699403f994836da202b9741" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38531 } }
I20260929 01:06:31.175395 22040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a8d1015f699403f994836da202b9741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:31.175472 22040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a8d1015f699403f994836da202b9741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a8d1015f699403f994836da202b9741, State: Initialized, Role: FOLLOWER
I20260929 01:06:31.175971 22040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a8d1015f699403f994836da202b9741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8d1015f699403f994836da202b9741" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38531 } }
I20260929 01:06:31.176141 22040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a8d1015f699403f994836da202b9741 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:31.176209 22040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a8d1015f699403f994836da202b9741 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:31.176303 22040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a8d1015f699403f994836da202b9741 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:31.177165 22040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a8d1015f699403f994836da202b9741 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8d1015f699403f994836da202b9741" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38531 } }
I20260929 01:06:31.177525 22040 leader_election.cc:302] T 00000000000000000000000000000000 P 8a8d1015f699403f994836da202b9741 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a8d1015f699403f994836da202b9741; no voters:
I20260929 01:06:31.177814 22040 leader_election.cc:288] T 00000000000000000000000000000000 P 8a8d1015f699403f994836da202b9741 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:31.177910 22050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a8d1015f699403f994836da202b9741 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:31.178140 22050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a8d1015f699403f994836da202b9741 [term 1 LEADER]: Becoming Leader. State: Replica: 8a8d1015f699403f994836da202b9741, State: Running, Role: LEADER
I20260929 01:06:31.178501 22050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a8d1015f699403f994836da202b9741 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8d1015f699403f994836da202b9741" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38531 } }
I20260929 01:06:31.179040 22040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a8d1015f699403f994836da202b9741 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:31.179731 22050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a8d1015f699403f994836da202b9741 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a8d1015f699403f994836da202b9741. Latest consensus state: current_term: 1 leader_uuid: "8a8d1015f699403f994836da202b9741" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8d1015f699403f994836da202b9741" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38531 } } }
I20260929 01:06:31.179912 22050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a8d1015f699403f994836da202b9741 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:31.180325 22059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:31.180496 22051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a8d1015f699403f994836da202b9741 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a8d1015f699403f994836da202b9741" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8d1015f699403f994836da202b9741" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38531 } } }
I20260929 01:06:31.180673 22051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a8d1015f699403f994836da202b9741 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:31.183734 22059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:31.185586 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:31.176056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47851 (local address 127.0.98.62:38531)
0929 01:06:31.176171 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:31.176175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:31.176392 (+ 217us) server_negotiation.cc:407] Connection header received
0929 01:06:31.176527 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:31.176530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:31.176574 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:31.176653 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:31.177278 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:31.177911 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:31.179061 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:31.179372 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:31.179463 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:31.181199 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:31.181202 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:31.181205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:31.181228 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:31.182918 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:31.183525 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:31.183529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:31.183530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:31.183610 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:31.184280 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:31.184283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:31.184284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:31.184479 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:31.184639 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:31.185333 (+ 694us) server_negotiation.cc:299] Negotiation successful
0929 01:06:31.185421 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 01:06:31.190538 22059 catalog_manager.cc:1347] Generated new cluster ID: ed77c7ce7970489bb36a67a4007535e5
I20260929 01:06:31.190610 22059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:31.195374 22059 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:31.196350 22059 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:31.209403 22059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a8d1015f699403f994836da202b9741: Generated new TSK 0
I20260929 01:06:31.210052 22059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:31.222179 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 21972
W20260929 01:06:31.227465 21957 connection.cc:535] client connection to 127.0.98.62:38531 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:38531): Connection reset by peer (error 104)
2026-09-29T01:06:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:31 dist-test-slave-7zdl krb5kdc[22071](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:31 dist-test-slave-7zdl krb5kdc[22071](info): set up 2 sockets
Sep 29 01:06:31 dist-test-slave-7zdl krb5kdc[22071](info): commencing operation
krb5kdc: starting...
W20260929 01:06:32.247038 392 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:06:32 dist-test-slave-7zdl krb5kdc[22071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:06:32.369349 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34569
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:32.475819 22084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:32.476049 22084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:32.476089 22084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:32.479941 22084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:32.479988 22084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:32.480006 22084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:32.480022 22084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:32.480037 22084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:32.485086 22084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34569
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:32.485985 22084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:32.486900 22084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:32.494666 22092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:32.494827 22084 server_base.cc:1034] running on GCE node
W20260929 01:06:32.494673 22090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:32.494680 22089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:32.495566 22084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:32.496603 22084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:32.497759 22084 hybrid_clock.cc:648] HybridClock initialized: now 1790643992497744 us; error 30 us; skew 500 ppm
Sep 29 01:06:32 dist-test-slave-7zdl krb5kdc[22071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:32.500388 22084 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:06:32.501200 22084 webserver.cc:460] Webserver started at http://127.0.98.62:43539/ using document root <none> and password file <none>
I20260929 01:06:32.501727 22084 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:32.501785 22084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:32.502030 22084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:32.504038 22084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "c46b4c0695ec435e8f970ac923c13b6c"
format_stamp: "Formatted at 2026-09-29 01:06:32 on dist-test-slave-7zdl"
server_key: "1cc823d21c1c7e89956e0ba750298040"
server_key_iv: "9efb1bff092f68fc4e6bcd1d132bb45b"
server_key_version: "encryptionkey@0"
I20260929 01:06:32.504585 22084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "c46b4c0695ec435e8f970ac923c13b6c"
format_stamp: "Formatted at 2026-09-29 01:06:32 on dist-test-slave-7zdl"
server_key: "1cc823d21c1c7e89956e0ba750298040"
server_key_iv: "9efb1bff092f68fc4e6bcd1d132bb45b"
server_key_version: "encryptionkey@0"
I20260929 01:06:32.508033 22084 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:06:32.510324 22099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:32.511636 22084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:06:32.511796 22084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "c46b4c0695ec435e8f970ac923c13b6c"
format_stamp: "Formatted at 2026-09-29 01:06:32 on dist-test-slave-7zdl"
server_key: "1cc823d21c1c7e89956e0ba750298040"
server_key_iv: "9efb1bff092f68fc4e6bcd1d132bb45b"
server_key_version: "encryptionkey@0"
I20260929 01:06:32.511919 22084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:32.527907 22084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:32.531005 22084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:32.531229 22084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:32.538115 22084 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34569
I20260929 01:06:32.538142 22151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34569 every 8 connection(s)
I20260929 01:06:32.539058 22084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:32.541738 22152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:32.545115 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 22084
I20260929 01:06:32.545348 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:32.545629 392 external_mini_cluster.cc:1428] Setting key 36e209f8363654a3bf44218d7a03aa6a
I20260929 01:06:32.547747 22152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c46b4c0695ec435e8f970ac923c13b6c: Bootstrap starting.
I20260929 01:06:32.550217 22152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c46b4c0695ec435e8f970ac923c13b6c: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:32.550987 22152 log.cc:824] T 00000000000000000000000000000000 P c46b4c0695ec435e8f970ac923c13b6c: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:32.552711 22152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c46b4c0695ec435e8f970ac923c13b6c: No bootstrap required, opened a new log
I20260929 01:06:32.555383 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:32.547119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49909 (local address 127.0.98.62:34569)
0929 01:06:32.547603 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:32.547625 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:32.547668 (+ 43us) server_negotiation.cc:407] Connection header received
0929 01:06:32.548610 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:32.548635 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:32.549087 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:32.549491 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:32.550204 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:32.551354 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:32.552241 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:32.552708 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:32.552881 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:32.553777 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:32.553782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:32.553793 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:32.554161 (+ 368us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:32.554210 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:32.554231 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:32.554368 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:32.554545 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 01:06:32.554675 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":132,"threads_started":1}
I20260929 01:06:32.556574 22152 raft_consensus.cc:348] T 00000000000000000000000000000000 P c46b4c0695ec435e8f970ac923c13b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46b4c0695ec435e8f970ac923c13b6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34569 } }
I20260929 01:06:32.556795 22152 raft_consensus.cc:374] T 00000000000000000000000000000000 P c46b4c0695ec435e8f970ac923c13b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:32.556887 22152 raft_consensus.cc:729] T 00000000000000000000000000000000 P c46b4c0695ec435e8f970ac923c13b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c46b4c0695ec435e8f970ac923c13b6c, State: Initialized, Role: FOLLOWER
I20260929 01:06:32.557348 22152 consensus_queue.cc:260] T 00000000000000000000000000000000 P c46b4c0695ec435e8f970ac923c13b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46b4c0695ec435e8f970ac923c13b6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34569 } }
I20260929 01:06:32.557503 22152 raft_consensus.cc:388] T 00000000000000000000000000000000 P c46b4c0695ec435e8f970ac923c13b6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:32.557574 22152 raft_consensus.cc:482] T 00000000000000000000000000000000 P c46b4c0695ec435e8f970ac923c13b6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:32.557665 22152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c46b4c0695ec435e8f970ac923c13b6c [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:32.558305 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:32.558640 22152 raft_consensus.cc:504] T 00000000000000000000000000000000 P c46b4c0695ec435e8f970ac923c13b6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46b4c0695ec435e8f970ac923c13b6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34569 } }
I20260929 01:06:32.558992 22152 leader_election.cc:302] T 00000000000000000000000000000000 P c46b4c0695ec435e8f970ac923c13b6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c46b4c0695ec435e8f970ac923c13b6c; no voters:
I20260929 01:06:32.559312 22152 leader_election.cc:288] T 00000000000000000000000000000000 P c46b4c0695ec435e8f970ac923c13b6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:32.559404 22157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c46b4c0695ec435e8f970ac923c13b6c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:32.559599 22157 raft_consensus.cc:686] T 00000000000000000000000000000000 P c46b4c0695ec435e8f970ac923c13b6c [term 1 LEADER]: Becoming Leader. State: Replica: c46b4c0695ec435e8f970ac923c13b6c, State: Running, Role: LEADER
I20260929 01:06:32.559901 22157 consensus_queue.cc:237] T 00000000000000000000000000000000 P c46b4c0695ec435e8f970ac923c13b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46b4c0695ec435e8f970ac923c13b6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34569 } }
I20260929 01:06:32.560561 22152 sys_catalog.cc:564] T 00000000000000000000000000000000 P c46b4c0695ec435e8f970ac923c13b6c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:32.561040 22157 sys_catalog.cc:455] T 00000000000000000000000000000000 P c46b4c0695ec435e8f970ac923c13b6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c46b4c0695ec435e8f970ac923c13b6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46b4c0695ec435e8f970ac923c13b6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34569 } } }
I20260929 01:06:32.561203 22157 sys_catalog.cc:458] T 00000000000000000000000000000000 P c46b4c0695ec435e8f970ac923c13b6c [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:32.561478 22157 sys_catalog.cc:455] T 00000000000000000000000000000000 P c46b4c0695ec435e8f970ac923c13b6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c46b4c0695ec435e8f970ac923c13b6c. Latest consensus state: current_term: 1 leader_uuid: "c46b4c0695ec435e8f970ac923c13b6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c46b4c0695ec435e8f970ac923c13b6c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34569 } } }
I20260929 01:06:32.561630 22157 sys_catalog.cc:458] T 00000000000000000000000000000000 P c46b4c0695ec435e8f970ac923c13b6c [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:32.561934 22169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:32.565017 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:32.560253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43549 (local address 127.0.98.62:34569)
0929 01:06:32.560362 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:32.560369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:32.560700 (+ 331us) server_negotiation.cc:407] Connection header received
0929 01:06:32.560792 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:32.560795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:32.560832 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:32.560924 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:32.561174 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:32.561908 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:32.563099 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:32.563419 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:32.563540 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:32.564394 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:32.564397 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:32.564400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:32.564412 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:32.564438 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:32.564451 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:32.564545 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:32.564854 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 01:06:32.564885 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 01:06:32.565526 22169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:32.570420 22169 catalog_manager.cc:1347] Generated new cluster ID: 6ba95bc63d994ef4a03cbf57bbcada75
I20260929 01:06:32.570479 22169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:32.594081 22169 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:32.595084 22169 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:32.603456 22169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c46b4c0695ec435e8f970ac923c13b6c: Generated new TSK 0
I20260929 01:06:32.604130 22169 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:32.611186 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 22084
W20260929 01:06:32.616497 22069 connection.cc:535] client connection to 127.0.98.62:34569 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:34569): Connection reset by peer (error 104)
2026-09-29T01:06:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:32 dist-test-slave-7zdl krb5kdc[22183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:32 dist-test-slave-7zdl krb5kdc[22183](info): set up 2 sockets
Sep 29 01:06:32 dist-test-slave-7zdl krb5kdc[22183](info): commencing operation
krb5kdc: starting...
W20260929 01:06:33.723042 392 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:06:33 dist-test-slave-7zdl krb5kdc[22183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:06:33.842314 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46557
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:33.951261 22196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:33.951522 22196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:33.951592 22196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:33.955545 22196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:33.955629 22196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:33.955667 22196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:33.955842 22196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:33.955888 22196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:33.961138 22196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46557
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:33.962144 22196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:33.963209 22196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:33.971422 22204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:33.971379 22202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:33.971395 22201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:33.971593 22196 server_base.cc:1034] running on GCE node
I20260929 01:06:33.972162 22196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:33.973220 22196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:33.974381 22196 hybrid_clock.cc:648] HybridClock initialized: now 1790643993974371 us; error 26 us; skew 500 ppm
Sep 29 01:06:33 dist-test-slave-7zdl krb5kdc[22183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:33.977129 22196 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:06:33.977949 22196 webserver.cc:460] Webserver started at http://127.0.98.62:34559/ using document root <none> and password file <none>
I20260929 01:06:33.978493 22196 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:33.978583 22196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:33.978849 22196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:33.980792 22196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "3af01237ca05406c84cbe414a3fc25ad"
format_stamp: "Formatted at 2026-09-29 01:06:33 on dist-test-slave-7zdl"
server_key: "ad1ccae883f4fe2364dbec1da1609add"
server_key_iv: "fce7a0edb45fffab423c0927cf4dd066"
server_key_version: "encryptionkey@0"
I20260929 01:06:33.981336 22196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "3af01237ca05406c84cbe414a3fc25ad"
format_stamp: "Formatted at 2026-09-29 01:06:33 on dist-test-slave-7zdl"
server_key: "ad1ccae883f4fe2364dbec1da1609add"
server_key_iv: "fce7a0edb45fffab423c0927cf4dd066"
server_key_version: "encryptionkey@0"
I20260929 01:06:33.984906 22196 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:06:33.987211 22211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:33.988373 22196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:06:33.988529 22196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "3af01237ca05406c84cbe414a3fc25ad"
format_stamp: "Formatted at 2026-09-29 01:06:33 on dist-test-slave-7zdl"
server_key: "ad1ccae883f4fe2364dbec1da1609add"
server_key_iv: "fce7a0edb45fffab423c0927cf4dd066"
server_key_version: "encryptionkey@0"
I20260929 01:06:33.988636 22196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:33.992905 22196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:33.995914 22196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:33.996105 22196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:34.002885 22196 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46557
I20260929 01:06:34.002913 22263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46557 every 8 connection(s)
I20260929 01:06:34.003778 22196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:34.006454 22264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:34.007895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 22196
I20260929 01:06:34.008044 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:34.008363 392 external_mini_cluster.cc:1428] Setting key 8736e0c2a9ded4094ef1c6378b4ab0f7
I20260929 01:06:34.012562 22264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3af01237ca05406c84cbe414a3fc25ad: Bootstrap starting.
I20260929 01:06:34.013350 22267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:34.009753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45119 (local address 127.0.98.62:46557)
0929 01:06:34.010174 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:34.010189 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:34.010223 (+ 34us) server_negotiation.cc:407] Connection header received
0929 01:06:34.010900 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:34.010917 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:34.011176 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:34.011490 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:34.012233 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:34.012238 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:34.012248 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:34.012258 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:34.012314 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:34.012332 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:34.012471 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:34.012537 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 01:06:34.012699 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":122,"threads_started":1}
I20260929 01:06:34.014731 22264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3af01237ca05406c84cbe414a3fc25ad: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:34.015597 22264 log.cc:824] T 00000000000000000000000000000000 P 3af01237ca05406c84cbe414a3fc25ad: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:34.015913 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:34.017467 22264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3af01237ca05406c84cbe414a3fc25ad: No bootstrap required, opened a new log
I20260929 01:06:34.019457 22267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:34.017844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34795 (local address 127.0.98.62:46557)
0929 01:06:34.017954 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:34.017958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:34.018121 (+ 163us) server_negotiation.cc:407] Connection header received
0929 01:06:34.018231 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:34.018234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:34.018273 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:34.018349 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:34.019059 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:34.019069 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:34.019072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:34.019082 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:34.019121 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:34.019130 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:34.019204 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:34.019308 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 01:06:34.019344 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 01:06:34.021533 22264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3af01237ca05406c84cbe414a3fc25ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af01237ca05406c84cbe414a3fc25ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46557 } }
I20260929 01:06:34.021745 22264 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3af01237ca05406c84cbe414a3fc25ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:34.021826 22264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3af01237ca05406c84cbe414a3fc25ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af01237ca05406c84cbe414a3fc25ad, State: Initialized, Role: FOLLOWER
I20260929 01:06:34.022271 22264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3af01237ca05406c84cbe414a3fc25ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af01237ca05406c84cbe414a3fc25ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46557 } }
I20260929 01:06:34.022423 22264 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3af01237ca05406c84cbe414a3fc25ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:34.022507 22264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3af01237ca05406c84cbe414a3fc25ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:34.022634 22264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3af01237ca05406c84cbe414a3fc25ad [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:34.023514 22264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3af01237ca05406c84cbe414a3fc25ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af01237ca05406c84cbe414a3fc25ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46557 } }
I20260929 01:06:34.023844 22264 leader_election.cc:302] T 00000000000000000000000000000000 P 3af01237ca05406c84cbe414a3fc25ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3af01237ca05406c84cbe414a3fc25ad; no voters:
I20260929 01:06:34.024121 22264 leader_election.cc:288] T 00000000000000000000000000000000 P 3af01237ca05406c84cbe414a3fc25ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:34.024186 22274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3af01237ca05406c84cbe414a3fc25ad [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:34.024497 22274 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3af01237ca05406c84cbe414a3fc25ad [term 1 LEADER]: Becoming Leader. State: Replica: 3af01237ca05406c84cbe414a3fc25ad, State: Running, Role: LEADER
I20260929 01:06:34.024835 22274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3af01237ca05406c84cbe414a3fc25ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af01237ca05406c84cbe414a3fc25ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46557 } }
I20260929 01:06:34.025450 22264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3af01237ca05406c84cbe414a3fc25ad [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:34.026592 22276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3af01237ca05406c84cbe414a3fc25ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3af01237ca05406c84cbe414a3fc25ad. Latest consensus state: current_term: 1 leader_uuid: "3af01237ca05406c84cbe414a3fc25ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af01237ca05406c84cbe414a3fc25ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46557 } } }
I20260929 01:06:34.026592 22275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3af01237ca05406c84cbe414a3fc25ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3af01237ca05406c84cbe414a3fc25ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af01237ca05406c84cbe414a3fc25ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46557 } } }
I20260929 01:06:34.026737 22276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3af01237ca05406c84cbe414a3fc25ad [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:34.026786 22275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3af01237ca05406c84cbe414a3fc25ad [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:34.027211 22283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:34.029923 22283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:34.035552 22283 catalog_manager.cc:1347] Generated new cluster ID: 9855c34ae8db4a70b8eca9162e241ede
I20260929 01:06:34.035622 22283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:34.050043 22283 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:34.050341 22283 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:34.059315 22283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3af01237ca05406c84cbe414a3fc25ad: Generated new TSK 0
I20260929 01:06:34.059921 22283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:34.066510 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 22196
2026-09-29T01:06:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1455 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:34 dist-test-slave-7zdl krb5kdc[22295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:34 dist-test-slave-7zdl krb5kdc[22295](info): set up 2 sockets
Sep 29 01:06:34 dist-test-slave-7zdl krb5kdc[22295](info): commencing operation
krb5kdc: starting...
W20260929 01:06:35.091110 392 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:06:35 dist-test-slave-7zdl krb5kdc[22295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:06:35.215977 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45483
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:35.325625 22308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:35.325887 22308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:35.325966 22308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:35.329989 22308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:35.330049 22308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:35.330070 22308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:35.330129 22308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:35.330165 22308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:35.335527 22308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:35.336562 22308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:35.337687 22308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:35.346268 22314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:35.346298 22313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:35.346298 22316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:35.346637 22308 server_base.cc:1034] running on GCE node
I20260929 01:06:35.347200 22308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:35.348258 22308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:35.349455 22308 hybrid_clock.cc:648] HybridClock initialized: now 1790643995349443 us; error 54 us; skew 500 ppm
Sep 29 01:06:35 dist-test-slave-7zdl krb5kdc[22295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:35.352172 22308 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:06:35.353106 22308 webserver.cc:460] Webserver started at http://127.0.98.62:37181/ using document root <none> and password file <none>
I20260929 01:06:35.353663 22308 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:35.353749 22308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:35.353976 22308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:35.356033 22308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "9ae7bfeebf5c42a18f896fc394eefe29"
format_stamp: "Formatted at 2026-09-29 01:06:35 on dist-test-slave-7zdl"
server_key: "3f88f1074e6c43b310477e53f44b1c7e"
server_key_iv: "6f34cd97f9d5bdd26c911aac47b775e3"
server_key_version: "encryptionkey@0"
I20260929 01:06:35.356583 22308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "9ae7bfeebf5c42a18f896fc394eefe29"
format_stamp: "Formatted at 2026-09-29 01:06:35 on dist-test-slave-7zdl"
server_key: "3f88f1074e6c43b310477e53f44b1c7e"
server_key_iv: "6f34cd97f9d5bdd26c911aac47b775e3"
server_key_version: "encryptionkey@0"
I20260929 01:06:35.360191 22308 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:06:35.362612 22323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:35.363888 22308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:06:35.364037 22308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "9ae7bfeebf5c42a18f896fc394eefe29"
format_stamp: "Formatted at 2026-09-29 01:06:35 on dist-test-slave-7zdl"
server_key: "3f88f1074e6c43b310477e53f44b1c7e"
server_key_iv: "6f34cd97f9d5bdd26c911aac47b775e3"
server_key_version: "encryptionkey@0"
I20260929 01:06:35.364153 22308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:35.385622 22308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:35.388664 22308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:35.388872 22308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:35.395777 22308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45483
I20260929 01:06:35.395802 22375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45483 every 8 connection(s)
I20260929 01:06:35.396693 22308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:35.399278 22376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:35.402208 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 22308
I20260929 01:06:35.402366 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:35.402669 392 external_mini_cluster.cc:1428] Setting key 15a2db2d644669993a6d5479de613654
I20260929 01:06:35.405278 22376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ae7bfeebf5c42a18f896fc394eefe29: Bootstrap starting.
I20260929 01:06:35.407511 22376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ae7bfeebf5c42a18f896fc394eefe29: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:35.408218 22376 log.cc:824] T 00000000000000000000000000000000 P 9ae7bfeebf5c42a18f896fc394eefe29: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:35.410115 22376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ae7bfeebf5c42a18f896fc394eefe29: No bootstrap required, opened a new log
Sep 29 01:06:35 dist-test-slave-7zdl krb5kdc[22295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:06:35.413453 22376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ae7bfeebf5c42a18f896fc394eefe29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae7bfeebf5c42a18f896fc394eefe29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45483 } }
I20260929 01:06:35.413736 22376 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ae7bfeebf5c42a18f896fc394eefe29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:35.413817 22376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ae7bfeebf5c42a18f896fc394eefe29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae7bfeebf5c42a18f896fc394eefe29, State: Initialized, Role: FOLLOWER
I20260929 01:06:35.414243 22376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ae7bfeebf5c42a18f896fc394eefe29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae7bfeebf5c42a18f896fc394eefe29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45483 } }
I20260929 01:06:35.414388 22376 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ae7bfeebf5c42a18f896fc394eefe29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:35.414470 22376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ae7bfeebf5c42a18f896fc394eefe29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:35.414587 22376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ae7bfeebf5c42a18f896fc394eefe29 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:35.415439 22376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ae7bfeebf5c42a18f896fc394eefe29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae7bfeebf5c42a18f896fc394eefe29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45483 } }
I20260929 01:06:35.415787 22376 leader_election.cc:302] T 00000000000000000000000000000000 P 9ae7bfeebf5c42a18f896fc394eefe29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ae7bfeebf5c42a18f896fc394eefe29; no voters:
I20260929 01:06:35.416082 22376 leader_election.cc:288] T 00000000000000000000000000000000 P 9ae7bfeebf5c42a18f896fc394eefe29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:35.416185 22381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ae7bfeebf5c42a18f896fc394eefe29 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:35.416437 22381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ae7bfeebf5c42a18f896fc394eefe29 [term 1 LEADER]: Becoming Leader. State: Replica: 9ae7bfeebf5c42a18f896fc394eefe29, State: Running, Role: LEADER
I20260929 01:06:35.416798 22381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ae7bfeebf5c42a18f896fc394eefe29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae7bfeebf5c42a18f896fc394eefe29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45483 } }
I20260929 01:06:35.417331 22376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ae7bfeebf5c42a18f896fc394eefe29 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:35.418009 22379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:35.404055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49219 (local address 127.0.98.62:45483)
0929 01:06:35.404507 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:35.404522 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:35.404557 (+ 35us) server_negotiation.cc:407] Connection header received
0929 01:06:35.405160 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:35.405184 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:35.405590 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:35.405977 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:35.406781 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:35.407871 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:35.408765 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:35.409220 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:35.409385 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:35.412225 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:35.412230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:35.412242 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:35.412268 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:35.414731 (+ 2463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:35.415183 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:35.415189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:35.415194 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:35.415323 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:35.415597 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:35.415600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:35.415601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:35.416078 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:35.416294 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:35.416729 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 01:06:35.417008 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":134,"threads_started":1}
I20260929 01:06:35.419509 22382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae7bfeebf5c42a18f896fc394eefe29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ae7bfeebf5c42a18f896fc394eefe29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae7bfeebf5c42a18f896fc394eefe29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45483 } } }
I20260929 01:06:35.420063 22382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae7bfeebf5c42a18f896fc394eefe29 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:35.419695 22383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae7bfeebf5c42a18f896fc394eefe29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ae7bfeebf5c42a18f896fc394eefe29. Latest consensus state: current_term: 1 leader_uuid: "9ae7bfeebf5c42a18f896fc394eefe29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae7bfeebf5c42a18f896fc394eefe29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45483 } } }
I20260929 01:06:35.420384 22383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae7bfeebf5c42a18f896fc394eefe29 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:35.421216 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 01:06:35.422482 22395 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ae7bfeebf5c42a18f896fc394eefe29: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:06:35.422580 22395 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:06:35.422665 22397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:35.423568 22397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:35.429417 22397 catalog_manager.cc:1347] Generated new cluster ID: de8293017e1644a9bc09cce908039a24
I20260929 01:06:35.429488 22397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:35.434718 22379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:35.425915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59485 (local address 127.0.98.62:45483)
0929 01:06:35.426009 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:35.426014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:35.426287 (+ 273us) server_negotiation.cc:407] Connection header received
0929 01:06:35.426387 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:35.426390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:35.426438 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:35.426524 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:35.427342 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:35.428167 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:35.429347 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:35.429694 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:35.429807 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:35.431407 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:35.431411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:35.431414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:35.431442 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:35.433090 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:35.433541 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:35.433545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:35.433547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:35.433632 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:35.433892 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:35.433895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:35.433897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:35.434108 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:35.434232 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:35.434446 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 01:06:35.434560 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 01:06:35.452917 22397 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:35.453796 22397 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:35.460534 22397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ae7bfeebf5c42a18f896fc394eefe29: Generated new TSK 0
I20260929 01:06:35.461146 22397 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:35.468365 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 22308
2026-09-29T01:06:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:35 dist-test-slave-7zdl krb5kdc[22407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:35 dist-test-slave-7zdl krb5kdc[22407](info): set up 2 sockets
Sep 29 01:06:35 dist-test-slave-7zdl krb5kdc[22407](info): commencing operation
krb5kdc: starting...
W20260929 01:06:36.507551 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:06:36 dist-test-slave-7zdl krb5kdc[22407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:06:36.631050 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41209
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:36.741622 22420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:36.741904 22420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:36.741988 22420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:36.746002 22420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:36.746088 22420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:36.746124 22420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:36.746184 22420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:36.746223 22420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:36.751449 22420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:36.752521 22420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:36.753533 22420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:36.762043 22428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:36.762025 22426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:36.762027 22425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:36.762074 22420 server_base.cc:1034] running on GCE node
I20260929 01:06:36.762878 22420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:36.763947 22420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:36.765134 22420 hybrid_clock.cc:648] HybridClock initialized: now 1790643996765118 us; error 32 us; skew 500 ppm
Sep 29 01:06:36 dist-test-slave-7zdl krb5kdc[22407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:36.773563 22420 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:06:36.774446 22420 webserver.cc:460] Webserver started at http://127.0.98.62:44015/ using document root <none> and password file <none>
I20260929 01:06:36.775066 22420 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:36.775161 22420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:36.775393 22420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:36.777424 22420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "0c324965ab11449995744cc1b58a4365"
format_stamp: "Formatted at 2026-09-29 01:06:36 on dist-test-slave-7zdl"
server_key: "8c075f671a30058fb134cb5260dda282"
server_key_iv: "e315cd0fb30ae7dc84c53fa862437a9d"
server_key_version: "encryptionkey@0"
I20260929 01:06:36.777971 22420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "0c324965ab11449995744cc1b58a4365"
format_stamp: "Formatted at 2026-09-29 01:06:36 on dist-test-slave-7zdl"
server_key: "8c075f671a30058fb134cb5260dda282"
server_key_iv: "e315cd0fb30ae7dc84c53fa862437a9d"
server_key_version: "encryptionkey@0"
I20260929 01:06:36.781488 22420 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260929 01:06:36.783831 22435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:36.784947 22420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:06:36.785079 22420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "0c324965ab11449995744cc1b58a4365"
format_stamp: "Formatted at 2026-09-29 01:06:36 on dist-test-slave-7zdl"
server_key: "8c075f671a30058fb134cb5260dda282"
server_key_iv: "e315cd0fb30ae7dc84c53fa862437a9d"
server_key_version: "encryptionkey@0"
I20260929 01:06:36.785192 22420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:36.800561 22420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:36.803232 22420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:36.803431 22420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:36.810274 22420 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41209
I20260929 01:06:36.810303 22487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41209 every 8 connection(s)
I20260929 01:06:36.811203 22420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:36.813722 22488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:36.816689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 22420
I20260929 01:06:36.816865 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:36.817154 392 external_mini_cluster.cc:1428] Setting key a62d754d301a2fa59b1ee1784af788a8
I20260929 01:06:36.819295 22488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c324965ab11449995744cc1b58a4365: Bootstrap starting.
I20260929 01:06:36.821511 22488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c324965ab11449995744cc1b58a4365: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:36.822371 22488 log.cc:824] T 00000000000000000000000000000000 P 0c324965ab11449995744cc1b58a4365: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:36.824468 22488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c324965ab11449995744cc1b58a4365: No bootstrap required, opened a new log
I20260929 01:06:36.826333 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:36.818513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58183 (local address 127.0.98.62:41209)
0929 01:06:36.818966 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:36.818981 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:36.819015 (+ 34us) server_negotiation.cc:407] Connection header received
0929 01:06:36.819682 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:36.819698 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:36.820022 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:36.820282 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:36.820659 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:36.821711 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:36.822951 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:36.823518 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:36.823643 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:36.824448 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:36.824453 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:36.824464 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:36.824880 (+ 416us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:36.824931 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:36.824955 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:36.825032 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:36.825222 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 01:06:36.825462 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":138,"threads_started":1}
I20260929 01:06:36.827468 22488 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c324965ab11449995744cc1b58a4365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c324965ab11449995744cc1b58a4365" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41209 } }
I20260929 01:06:36.827663 22488 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c324965ab11449995744cc1b58a4365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:36.827729 22488 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c324965ab11449995744cc1b58a4365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c324965ab11449995744cc1b58a4365, State: Initialized, Role: FOLLOWER
I20260929 01:06:36.828212 22488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c324965ab11449995744cc1b58a4365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c324965ab11449995744cc1b58a4365" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41209 } }
I20260929 01:06:36.828344 22488 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c324965ab11449995744cc1b58a4365 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:36.828461 22488 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c324965ab11449995744cc1b58a4365 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:36.828567 22488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c324965ab11449995744cc1b58a4365 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:36.829119 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:36.829526 22488 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c324965ab11449995744cc1b58a4365 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c324965ab11449995744cc1b58a4365" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41209 } }
I20260929 01:06:36.829843 22488 leader_election.cc:302] T 00000000000000000000000000000000 P 0c324965ab11449995744cc1b58a4365 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c324965ab11449995744cc1b58a4365; no voters:
I20260929 01:06:36.830145 22488 leader_election.cc:288] T 00000000000000000000000000000000 P 0c324965ab11449995744cc1b58a4365 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:36.830204 22493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c324965ab11449995744cc1b58a4365 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:36.830438 22493 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c324965ab11449995744cc1b58a4365 [term 1 LEADER]: Becoming Leader. State: Replica: 0c324965ab11449995744cc1b58a4365, State: Running, Role: LEADER
I20260929 01:06:36.830727 22493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c324965ab11449995744cc1b58a4365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c324965ab11449995744cc1b58a4365" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41209 } }
I20260929 01:06:36.831439 22488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c324965ab11449995744cc1b58a4365 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:36.831838 22493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c324965ab11449995744cc1b58a4365 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c324965ab11449995744cc1b58a4365. Latest consensus state: current_term: 1 leader_uuid: "0c324965ab11449995744cc1b58a4365" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c324965ab11449995744cc1b58a4365" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41209 } } }
I20260929 01:06:36.832000 22493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c324965ab11449995744cc1b58a4365 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:36.832204 22493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c324965ab11449995744cc1b58a4365 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c324965ab11449995744cc1b58a4365" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c324965ab11449995744cc1b58a4365" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41209 } } }
I20260929 01:06:36.832307 22493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c324965ab11449995744cc1b58a4365 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:36.832477 22503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:36.835593 22503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:36.835807 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:36.831140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50047 (local address 127.0.98.62:41209)
0929 01:06:36.831419 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:36.831426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:36.831440 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:06:36.831583 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:36.831586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:36.831626 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:36.831714 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:36.831970 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:36.832727 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:36.833814 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:36.834113 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:36.834203 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:36.835151 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:36.835154 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:36.835157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:36.835168 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:36.835193 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:36.835205 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:36.835266 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:36.835644 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 01:06:36.835695 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260929 01:06:36.841475 22503 catalog_manager.cc:1347] Generated new cluster ID: e5ed12e5cae842489af0ab06b0ab10a0
I20260929 01:06:36.841544 22503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:36.870594 22503 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:36.871656 22503 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:36.879971 22503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c324965ab11449995744cc1b58a4365: Generated new TSK 0
I20260929 01:06:36.880651 22503 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:36.888249 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 22420
2026-09-29T01:06:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1419 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:36 dist-test-slave-7zdl krb5kdc[22519](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:36 dist-test-slave-7zdl krb5kdc[22519](info): set up 2 sockets
Sep 29 01:06:36 dist-test-slave-7zdl krb5kdc[22519](info): commencing operation
krb5kdc: starting...
W20260929 01:06:37.927049 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:06:38 dist-test-slave-7zdl krb5kdc[22519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:06:38.050460 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39963
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:38.159029 22532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:38.159349 22532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:38.159427 22532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:38.163437 22532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:38.163544 22532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:38.163583 22532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:38.163621 22532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:38.163681 22532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:38.168942 22532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:38.170109 22532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:38.171247 22532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:38.179828 22538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:38.179828 22537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:38.179854 22540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:38.180184 22532 server_base.cc:1034] running on GCE node
I20260929 01:06:38.180771 22532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:38.181810 22532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:38.182965 22532 hybrid_clock.cc:648] HybridClock initialized: now 1790643998182932 us; error 45 us; skew 500 ppm
Sep 29 01:06:38 dist-test-slave-7zdl krb5kdc[22519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:38.185746 22532 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:06:38.186643 22532 webserver.cc:460] Webserver started at http://127.0.98.62:45433/ using document root <none> and password file <none>
I20260929 01:06:38.187254 22532 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:38.187347 22532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:38.187580 22532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:38.189576 22532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "f61e3084343e47dcb0e6db2429cf570c"
format_stamp: "Formatted at 2026-09-29 01:06:38 on dist-test-slave-7zdl"
server_key: "c8159ad69a676bd9600f08f13766130c"
server_key_iv: "94b36cf517962bbb706ccae0bc2ecf60"
server_key_version: "encryptionkey@0"
I20260929 01:06:38.190145 22532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "f61e3084343e47dcb0e6db2429cf570c"
format_stamp: "Formatted at 2026-09-29 01:06:38 on dist-test-slave-7zdl"
server_key: "c8159ad69a676bd9600f08f13766130c"
server_key_iv: "94b36cf517962bbb706ccae0bc2ecf60"
server_key_version: "encryptionkey@0"
I20260929 01:06:38.193737 22532 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 01:06:38.196177 22547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:38.197373 22532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:06:38.197513 22532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "f61e3084343e47dcb0e6db2429cf570c"
format_stamp: "Formatted at 2026-09-29 01:06:38 on dist-test-slave-7zdl"
server_key: "c8159ad69a676bd9600f08f13766130c"
server_key_iv: "94b36cf517962bbb706ccae0bc2ecf60"
server_key_version: "encryptionkey@0"
I20260929 01:06:38.197624 22532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:38.202073 22532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:38.205265 22532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:38.205449 22532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:38.212402 22532 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39963
I20260929 01:06:38.212426 22599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39963 every 8 connection(s)
I20260929 01:06:38.213301 22532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:38.216109 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 22532
I20260929 01:06:38.216257 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:38.216243 22600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:38.216601 392 external_mini_cluster.cc:1428] Setting key e23fb0fcb04d41f34a2522db1d4c3926
I20260929 01:06:38.222172 22603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:38.218012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33165 (local address 127.0.98.62:39963)
0929 01:06:38.218423 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:38.218442 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:38.218486 (+ 44us) server_negotiation.cc:407] Connection header received
0929 01:06:38.219331 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:38.219354 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:38.219708 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:38.220083 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:38.220811 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:38.220820 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:38.220835 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:38.220849 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:38.220910 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:38.220946 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:38.221081 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:38.221145 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 01:06:38.221284 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":109,"threads_started":1}
I20260929 01:06:38.223497 22600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61e3084343e47dcb0e6db2429cf570c: Bootstrap starting.
I20260929 01:06:38.225080 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:38.225941 22600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f61e3084343e47dcb0e6db2429cf570c: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:38.226707 22600 log.cc:824] T 00000000000000000000000000000000 P f61e3084343e47dcb0e6db2429cf570c: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:38.228561 22600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61e3084343e47dcb0e6db2429cf570c: No bootstrap required, opened a new log
I20260929 01:06:38.228875 22603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:38.227156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33281 (local address 127.0.98.62:39963)
0929 01:06:38.227283 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:38.227287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:38.227480 (+ 193us) server_negotiation.cc:407] Connection header received
0929 01:06:38.227629 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:38.227631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:38.227672 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:38.227750 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:38.228471 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:38.228474 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:38.228477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 01:06:38.228488 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:38.228514 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 01:06:38.228527 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:38.228618 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:38.228728 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 01:06:38.228766 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 01:06:38.232277 22600 raft_consensus.cc:348] T 00000000000000000000000000000000 P f61e3084343e47dcb0e6db2429cf570c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61e3084343e47dcb0e6db2429cf570c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39963 } }
I20260929 01:06:38.232503 22600 raft_consensus.cc:374] T 00000000000000000000000000000000 P f61e3084343e47dcb0e6db2429cf570c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:06:38.232539 22600 raft_consensus.cc:729] T 00000000000000000000000000000000 P f61e3084343e47dcb0e6db2429cf570c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f61e3084343e47dcb0e6db2429cf570c, State: Initialized, Role: FOLLOWER
I20260929 01:06:38.232893 22600 consensus_queue.cc:260] T 00000000000000000000000000000000 P f61e3084343e47dcb0e6db2429cf570c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61e3084343e47dcb0e6db2429cf570c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39963 } }
I20260929 01:06:38.232997 22600 raft_consensus.cc:388] T 00000000000000000000000000000000 P f61e3084343e47dcb0e6db2429cf570c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:06:38.233067 22600 raft_consensus.cc:482] T 00000000000000000000000000000000 P f61e3084343e47dcb0e6db2429cf570c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:06:38.233187 22600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f61e3084343e47dcb0e6db2429cf570c [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:38.234093 22600 raft_consensus.cc:504] T 00000000000000000000000000000000 P f61e3084343e47dcb0e6db2429cf570c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61e3084343e47dcb0e6db2429cf570c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39963 } }
I20260929 01:06:38.234501 22600 leader_election.cc:302] T 00000000000000000000000000000000 P f61e3084343e47dcb0e6db2429cf570c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f61e3084343e47dcb0e6db2429cf570c; no voters:
I20260929 01:06:38.234874 22600 leader_election.cc:288] T 00000000000000000000000000000000 P f61e3084343e47dcb0e6db2429cf570c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:06:38.234963 22610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f61e3084343e47dcb0e6db2429cf570c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:38.235131 22610 raft_consensus.cc:686] T 00000000000000000000000000000000 P f61e3084343e47dcb0e6db2429cf570c [term 1 LEADER]: Becoming Leader. State: Replica: f61e3084343e47dcb0e6db2429cf570c, State: Running, Role: LEADER
I20260929 01:06:38.235424 22610 consensus_queue.cc:237] T 00000000000000000000000000000000 P f61e3084343e47dcb0e6db2429cf570c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61e3084343e47dcb0e6db2429cf570c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39963 } }
I20260929 01:06:38.236164 22600 sys_catalog.cc:564] T 00000000000000000000000000000000 P f61e3084343e47dcb0e6db2429cf570c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:38.237161 22612 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61e3084343e47dcb0e6db2429cf570c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f61e3084343e47dcb0e6db2429cf570c. Latest consensus state: current_term: 1 leader_uuid: "f61e3084343e47dcb0e6db2429cf570c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61e3084343e47dcb0e6db2429cf570c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39963 } } }
I20260929 01:06:38.237278 22612 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61e3084343e47dcb0e6db2429cf570c [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:38.237607 22611 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61e3084343e47dcb0e6db2429cf570c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f61e3084343e47dcb0e6db2429cf570c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f61e3084343e47dcb0e6db2429cf570c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39963 } } }
I20260929 01:06:38.237725 22611 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61e3084343e47dcb0e6db2429cf570c [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:38.237651 22619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:38.241215 22619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:38.246665 22619 catalog_manager.cc:1347] Generated new cluster ID: 6139ae3542034036abeb05148b188b11
I20260929 01:06:38.246732 22619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:38.270653 22619 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:38.271045 22619 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:38.277385 22619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f61e3084343e47dcb0e6db2429cf570c: Generated new TSK 0
I20260929 01:06:38.278193 22619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 01:06:38.285094 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 22532
2026-09-29T01:06:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1396 ms)
[----------] 18 tests from AuthTokenIssuingTest (25326 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:38 dist-test-slave-7zdl krb5kdc[22631](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:38 dist-test-slave-7zdl krb5kdc[22631](info): set up 2 sockets
Sep 29 01:06:38 dist-test-slave-7zdl krb5kdc[22631](info): commencing operation
krb5kdc: starting...
W20260929 01:06:39.319164 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:06:39 dist-test-slave-7zdl krb5kdc[22631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:06:39.441644 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34837
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34837,127.0.98.61:35615,127.0.98.60:43981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:39.548130 22644 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 01:06:39.548454 22644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:39.548525 22644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:39.548604 22644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:39.552591 22644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:39.552686 22644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:39.552726 22644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:39.552757 22644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:39.552788 22644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:39.557914 22644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34837,127.0.98.61:35615,127.0.98.60:43981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:39.558992 22644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:39.560004 22644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:39.568266 22649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:39.568317 22650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:39.568288 22652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:39.568382 22644 server_base.cc:1034] running on GCE node
I20260929 01:06:39.569043 22644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:39.570137 22644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:39.571333 22644 hybrid_clock.cc:648] HybridClock initialized: now 1790643999571318 us; error 34 us; skew 500 ppm
Sep 29 01:06:39 dist-test-slave-7zdl krb5kdc[22631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:39.574222 22644 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:06:39.575398 22644 webserver.cc:460] Webserver started at http://127.0.98.62:34881/ using document root <none> and password file <none>
I20260929 01:06:39.576068 22644 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:39.576165 22644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:39.576447 22644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:39.579087 22644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "95d4d108de8e46fea136cd61ce55b8c4"
format_stamp: "Formatted at 2026-09-29 01:06:39 on dist-test-slave-7zdl"
server_key: "da75b7dbba9fc347177baa6e4937604c"
server_key_iv: "9051a7a0de6ddc97c41aafad46a80cf8"
server_key_version: "encryptionkey@0"
I20260929 01:06:39.579666 22644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "95d4d108de8e46fea136cd61ce55b8c4"
format_stamp: "Formatted at 2026-09-29 01:06:39 on dist-test-slave-7zdl"
server_key: "da75b7dbba9fc347177baa6e4937604c"
server_key_iv: "9051a7a0de6ddc97c41aafad46a80cf8"
server_key_version: "encryptionkey@0"
I20260929 01:06:39.583496 22644 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:06:39.585975 22659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:39.587234 22644 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:06:39.587406 22644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "95d4d108de8e46fea136cd61ce55b8c4"
format_stamp: "Formatted at 2026-09-29 01:06:39 on dist-test-slave-7zdl"
server_key: "da75b7dbba9fc347177baa6e4937604c"
server_key_iv: "9051a7a0de6ddc97c41aafad46a80cf8"
server_key_version: "encryptionkey@0"
I20260929 01:06:39.587517 22644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:39.609087 22644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:39.612025 22644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:39.612228 22644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:39.618856 22644 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34837
I20260929 01:06:39.619140 22711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34837 every 8 connection(s)
I20260929 01:06:39.619823 22644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:39.622491 22712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:39.626045 22712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:06:39.627542 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 22644
I20260929 01:06:39.627681 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:39.628023 392 external_mini_cluster.cc:1428] Setting key f05f9df190b5e96d3d518044631d4a66
Sep 29 01:06:39 dist-test-slave-7zdl krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260929 01:06:39.642345 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.628332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34837 (local address 127.0.0.1:47690)
0929 01:06:39.628861 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:39.628911 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:39.629750 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:39.630429 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:39.630434 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:39.630903 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:39.631158 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:39.631171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:39.632878 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:39.632883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:39.633927 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:39.633940 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:39.634618 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:39.634624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:39.634780 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:39.635394 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:39.635413 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:39.637144 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:39.639321 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:39.639329 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:39.639339 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:39.639585 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:39.639944 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:39.639949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:39.639951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:39.640062 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:39.640728 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:39.640737 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:39.640899 (+ 162us) client_negotiation.cc:769] Sending connection context
0929 01:06:39.641094 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 01:06:39.641533 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":99,"thread_start_us":153,"threads_started":1}
I20260929 01:06:39.642354 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.628333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47690 (local address 127.0.98.62:34837)
0929 01:06:39.628917 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:39.628929 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:39.629010 (+ 81us) server_negotiation.cc:407] Connection header received
0929 01:06:39.630048 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:39.630059 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:39.630231 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:39.630429 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:39.631314 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:39.632731 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:39.634075 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:39.634455 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:39.634570 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:39.637278 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:39.637285 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:39.637297 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:39.637324 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:39.639139 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:39.639694 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:39.639702 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:39.639705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:39.639858 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:39.640182 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:39.640188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:39.640193 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:39.640607 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:39.640713 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:39.641158 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 01:06:39.641433 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":458,"spinlock_wait_cycles":40704,"thread_start_us":150,"threads_started":1}
I20260929 01:06:39.644682 22712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35615 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:06:39.645421 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.645155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35615 (local address 127.0.0.1:60362)
0929 01:06:39.645274 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:39.645368 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 01:06:39.645602 22662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35615: connect: Connection refused (error 111)
I20260929 01:06:39.646394 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.646189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35615 (local address 127.0.0.1:60378)
0929 01:06:39.646294 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:39.646341 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 01:06:39.646544 22712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35615: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35615: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 01:06:39.657707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35615
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34837,127.0.98.61:35615,127.0.98.60:43981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:06:39.695785 22712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35615 } attempt: 1
I20260929 01:06:39.696743 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.696381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35615 (local address 127.0.0.1:60380)
0929 01:06:39.696569 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:39.696650 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 01:06:39.697584 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.697339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35615 (local address 127.0.0.1:60392)
0929 01:06:39.697463 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:39.697521 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 01:06:39.697742 22712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35615: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35615: connect: Connection refused (error 111)
I20260929 01:06:39.765903 22712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35615 } attempt: 2
I20260929 01:06:39.766752 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.766433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35615 (local address 127.0.0.1:60404)
0929 01:06:39.766595 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:39.766663 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 01:06:39.767534 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.767307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35615 (local address 127.0.0.1:60416)
0929 01:06:39.767414 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:39.767471 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 01:06:39.767716 22712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35615: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35615: connect: Connection refused (error 111)
W20260929 01:06:39.769457 22719 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 01:06:39.769734 22719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:39.769802 22719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:39.769871 22719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:39.773887 22719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:39.773965 22719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:39.774010 22719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:39.774066 22719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:39.774102 22719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:39.779315 22719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34837,127.0.98.61:35615,127.0.98.60:43981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:39.780323 22719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:39.781337 22719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:39.789568 22727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:39.789562 22724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:39.789611 22719 server_base.cc:1034] running on GCE node
W20260929 01:06:39.789562 22725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:39.790307 22719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:39.791465 22719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:39.792632 22719 hybrid_clock.cc:648] HybridClock initialized: now 1790643999792617 us; error 26 us; skew 500 ppm
Sep 29 01:06:39 dist-test-slave-7zdl krb5kdc[22631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:39.795423 22719 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 01:06:39.796222 22719 webserver.cc:460] Webserver started at http://127.0.98.61:40087/ using document root <none> and password file <none>
I20260929 01:06:39.796795 22719 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:39.796890 22719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:39.797114 22719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:39.799149 22719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-1/data/instance:
uuid: "925e3b9e25594922aa9849d068d600e0"
format_stamp: "Formatted at 2026-09-29 01:06:39 on dist-test-slave-7zdl"
server_key: "1b263738bd4cb48183c668bb3aa0dab4"
server_key_iv: "e7d71eb80086816adc0ad40348bf4864"
server_key_version: "encryptionkey@0"
I20260929 01:06:39.799687 22719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-1/wal/instance:
uuid: "925e3b9e25594922aa9849d068d600e0"
format_stamp: "Formatted at 2026-09-29 01:06:39 on dist-test-slave-7zdl"
server_key: "1b263738bd4cb48183c668bb3aa0dab4"
server_key_iv: "e7d71eb80086816adc0ad40348bf4864"
server_key_version: "encryptionkey@0"
I20260929 01:06:39.803288 22719 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:06:39.805958 22734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:39.807340 22719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:06:39.807495 22719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-1/wal
uuid: "925e3b9e25594922aa9849d068d600e0"
format_stamp: "Formatted at 2026-09-29 01:06:39 on dist-test-slave-7zdl"
server_key: "1b263738bd4cb48183c668bb3aa0dab4"
server_key_iv: "e7d71eb80086816adc0ad40348bf4864"
server_key_version: "encryptionkey@0"
I20260929 01:06:39.807607 22719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:39.822240 22719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:39.825404 22719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:39.825603 22719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:39.832295 22719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35615
I20260929 01:06:39.832566 22786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35615 every 8 connection(s)
I20260929 01:06:39.833232 22719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-1/data/info.pb
I20260929 01:06:39.833599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 22719
I20260929 01:06:39.833745 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-1/wal/instance
I20260929 01:06:39.834049 392 external_mini_cluster.cc:1428] Setting key 310c1d1297669eaba9ec4291108af09e
I20260929 01:06:39.836476 22787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:39.840776 22787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 29 01:06:39 dist-test-slave-7zdl krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:06:39.855933 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.842780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47694 (local address 127.0.98.62:34837)
0929 01:06:39.842963 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:39.842969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:39.843756 (+ 787us) server_negotiation.cc:407] Connection header received
0929 01:06:39.844952 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:39.844955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:39.845007 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:39.845085 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:39.846538 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:39.847381 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:39.849096 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:39.849523 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:39.849662 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:39.852082 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:39.852088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:39.852095 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:39.852132 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:39.853829 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:39.854314 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:39.854317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:39.854319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:39.854406 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:39.854731 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:39.854736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:39.854738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:39.855023 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:39.855149 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:39.855646 (+ 497us) server_negotiation.cc:299] Negotiation successful
0929 01:06:39.855800 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 01:06:39.856655 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.843130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34837 (local address 127.0.0.1:47694)
0929 01:06:39.843582 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:39.843633 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:39.844628 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:39.845258 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:39.845265 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:39.845697 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:39.846232 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:39.846246 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:39.847528 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:39.847534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:39.848977 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:39.848986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:39.849683 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:39.849686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:39.849835 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:39.850458 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:39.850473 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:39.851904 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:39.853952 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:39.853956 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:39.853967 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:39.854210 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:39.854504 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:39.854506 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:39.854508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:39.854636 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:39.855138 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:39.855143 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:39.855386 (+ 243us) client_negotiation.cc:769] Sending connection context
0929 01:06:39.855547 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 01:06:39.855784 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":130,"threads_started":1}
I20260929 01:06:39.858117 22787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35615 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:06:39.858901 22712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35615 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Sep 29 01:06:39 dist-test-slave-7zdl krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 01:06:39.864696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43981
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34837,127.0.98.61:35615,127.0.98.60:43981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 01:06:39 dist-test-slave-7zdl krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 01:06:39.868945 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.858682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35615 (local address 127.0.0.1:60426)
0929 01:06:39.858826 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:39.858841 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:39.858915 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:39.859621 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:39.859640 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:39.859785 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:39.859871 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:39.859876 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:39.861128 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:39.861134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:39.862012 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:39.862022 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:39.862604 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:39.862607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:39.862711 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:39.863228 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:39.863242 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:39.864744 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:39.867254 (+ 2510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:39.867259 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:39.867263 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:39.867542 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:39.867866 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:39.867870 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:39.867872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:39.867951 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:39.868479 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:39.868481 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:39.868536 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 01:06:39.868641 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 01:06:39.868756 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 01:06:39.868978 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.858841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60426 (local address 127.0.98.61:35615)
0929 01:06:39.859198 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:39.859206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:39.859223 (+ 17us) server_negotiation.cc:407] Connection header received
0929 01:06:39.859258 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:39.859262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:39.859468 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:39.859602 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:39.859988 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:39.861003 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:39.862118 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:39.862488 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:39.862594 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:39.864900 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:39.864912 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:39.864927 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:39.864954 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:39.867084 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:39.867661 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:39.867667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:39.867668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:39.867751 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:39.868047 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:39.868052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:39.868055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:39.868360 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:39.868435 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:39.868715 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 01:06:39.868836 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":150,"threads_started":1}
I20260929 01:06:39.870101 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.859366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35615 (local address 127.0.0.1:60432)
0929 01:06:39.859511 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:39.859527 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:39.859591 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:39.860091 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:39.860097 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:39.860233 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:39.860324 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:39.860329 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:39.861244 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:39.861247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:39.862448 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:39.862457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:39.863188 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:39.863194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:39.863339 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:39.863734 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:39.863753 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:39.865184 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:39.868312 (+ 3128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:39.868318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:39.868323 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:39.868591 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:39.868954 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:39.868959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:39.868964 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:39.869063 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:39.869581 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:39.869584 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:39.869680 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 01:06:39.869804 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 01:06:39.869930 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 01:06:39.870283 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.859456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60432 (local address 127.0.98.61:35615)
0929 01:06:39.859856 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:39.859862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:39.859870 (+ 8us) server_negotiation.cc:407] Connection header received
0929 01:06:39.859893 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:39.859895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:39.859924 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:39.860033 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:39.860435 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:39.861161 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:39.862633 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:39.863048 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:39.863158 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:39.865384 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:39.865391 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:39.865397 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:39.865847 (+ 450us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:39.868191 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:39.868720 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:39.868723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:39.868725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:39.868822 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:39.869193 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:39.869197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:39.869202 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:39.869425 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:39.869571 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:39.869881 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 01:06:39.870021 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"spinlock_wait_cycles":25344,"thread_start_us":99,"threads_started":1}
I20260929 01:06:39.871236 22787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:06:39.871572 22712 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:06:39.872256 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.872027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43981 (local address 127.0.0.1:33878)
0929 01:06:39.872138 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:39.872189 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 01:06:39.872403 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.872062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43981 (local address 127.0.0.1:33882)
0929 01:06:39.872235 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:39.872315 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260929 01:06:39.872623 22735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43981: connect: Connection refused (error 111)
I20260929 01:06:39.872961 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.872671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43981 (local address 127.0.0.1:33896)
0929 01:06:39.872849 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:39.872886 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260929 01:06:39.873162 22712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43981: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43981: connect: Connection refused (error 111)
I20260929 01:06:39.873508 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.873272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43981 (local address 127.0.0.1:33902)
0929 01:06:39.873377 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:39.873435 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 01:06:39.873662 22787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43981: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43981: connect: Connection refused (error 111)
I20260929 01:06:39.904354 22712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } attempt: 1
I20260929 01:06:39.905356 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.904988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43981 (local address 127.0.0.1:33904)
0929 01:06:39.905197 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:39.905263 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 01:06:39.906067 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.905875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43981 (local address 127.0.0.1:33920)
0929 01:06:39.905974 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:39.906016 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 01:06:39.906231 22712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43981: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43981: connect: Connection refused (error 111)
I20260929 01:06:39.922928 22787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } attempt: 1
I20260929 01:06:39.923810 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.923479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43981 (local address 127.0.0.1:33928)
0929 01:06:39.923647 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:39.923719 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 01:06:39.924561 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.924331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43981 (local address 127.0.0.1:33932)
0929 01:06:39.924443 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:39.924495 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 01:06:39.924738 22787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43981: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43981: connect: Connection refused (error 111)
W20260929 01:06:39.977653 22794 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 01:06:39.977912 22794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:39.977934 22794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:39.978019 22794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 01:06:39.981417 22712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } attempt: 2
I20260929 01:06:39.982262 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.981965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43981 (local address 127.0.0.1:33940)
0929 01:06:39.982125 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:39.982188 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 01:06:39.982281 22794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:39.982354 22794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:39.982411 22794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:39.982494 22794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:39.982544 22794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:39.983095 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.982846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43981 (local address 127.0.0.1:33954)
0929 01:06:39.982956 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:39.983046 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 01:06:39.983223 22712 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43981: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43981: connect: Connection refused (error 111)
I20260929 01:06:39.987914 22794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34837,127.0.98.61:35615,127.0.98.60:43981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:39.988938 22794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:39.989944 22794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:06:39.992940 22787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } attempt: 2
I20260929 01:06:39.993712 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.993439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43981 (local address 127.0.0.1:33964)
0929 01:06:39.993591 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:39.993646 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 01:06:39.994485 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:39.994255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43981 (local address 127.0.0.1:33970)
0929 01:06:39.994370 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:39.994422 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 01:06:39.994614 22787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43981: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43981: connect: Connection refused (error 111)
W20260929 01:06:39.998724 22803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:39.998682 22800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:39.998683 22801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:39.999039 22794 server_base.cc:1034] running on GCE node
I20260929 01:06:39.999604 22794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:40.000663 22794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:40.001858 22794 hybrid_clock.cc:648] HybridClock initialized: now 1790644000001841 us; error 26 us; skew 500 ppm
Sep 29 01:06:39 dist-test-slave-7zdl krb5kdc[22631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:40.004663 22794 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 01:06:40.005448 22794 webserver.cc:460] Webserver started at http://127.0.98.60:39367/ using document root <none> and password file <none>
I20260929 01:06:40.005949 22794 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:40.006004 22794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:40.006158 22794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:40.008148 22794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-2/data/instance:
uuid: "0e97fe2a6a4b4a7e9a824ad825735c43"
format_stamp: "Formatted at 2026-09-29 01:06:39 on dist-test-slave-7zdl"
server_key: "fac3497c6d81985cb82f07a055f18262"
server_key_iv: "14c583b8c0bd7a15e47bd80e32f7d4b5"
server_key_version: "encryptionkey@0"
I20260929 01:06:40.008641 22794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-2/wal/instance:
uuid: "0e97fe2a6a4b4a7e9a824ad825735c43"
format_stamp: "Formatted at 2026-09-29 01:06:39 on dist-test-slave-7zdl"
server_key: "fac3497c6d81985cb82f07a055f18262"
server_key_iv: "14c583b8c0bd7a15e47bd80e32f7d4b5"
server_key_version: "encryptionkey@0"
I20260929 01:06:40.012205 22794 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:06:40.014457 22810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:40.015692 22794 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:06:40.015872 22794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-2/wal
uuid: "0e97fe2a6a4b4a7e9a824ad825735c43"
format_stamp: "Formatted at 2026-09-29 01:06:39 on dist-test-slave-7zdl"
server_key: "fac3497c6d81985cb82f07a055f18262"
server_key_iv: "14c583b8c0bd7a15e47bd80e32f7d4b5"
server_key_version: "encryptionkey@0"
I20260929 01:06:40.015985 22794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:40.035444 22794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:40.038857 22794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:40.039000 22794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:40.045513 22794 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43981
I20260929 01:06:40.045806 22862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43981 every 8 connection(s)
I20260929 01:06:40.046500 22794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-2/data/info.pb
I20260929 01:06:40.049276 22863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:40.051544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 22794
I20260929 01:06:40.051646 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790643720886468-392-0/minicluster-data/master-2/wal/instance
I20260929 01:06:40.052006 392 external_mini_cluster.cc:1428] Setting key d0e9635647abb27692052d8a7fdba848
I20260929 01:06:40.053179 22863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:06:40 dist-test-slave-7zdl krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 01:06:40 dist-test-slave-7zdl krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 01:06:40.065083 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:40.054956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47696 (local address 127.0.98.62:34837)
0929 01:06:40.055096 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:40.055102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:40.055358 (+ 256us) server_negotiation.cc:407] Connection header received
0929 01:06:40.055579 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:40.055583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:40.055638 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:40.055682 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:40.056502 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.057238 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.058181 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.058463 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.058542 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:40.061178 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:40.061181 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:40.061183 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:40.061205 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:40.062926 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:40.063444 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:40.063449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:40.063450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:40.063538 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:40.063957 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:40.063960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:40.063962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:40.064204 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:40.064300 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:40.064781 (+ 481us) server_negotiation.cc:299] Negotiation successful
0929 01:06:40.064918 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 01:06:40.065871 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:40.054878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34837 (local address 127.0.0.1:47696)
0929 01:06:40.055236 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:40.055281 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:40.055461 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:40.055833 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:40.055837 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:40.056143 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:40.056362 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:40.056372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.057343 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.057345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:40.058062 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:40.058071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.058609 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.058612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:40.058783 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:40.059455 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:40.059479 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:40.061024 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:40.063026 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:40.063033 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:40.063040 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:40.063346 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:40.063629 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:40.063634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:40.063639 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:40.063827 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:40.064306 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:40.064312 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:40.064532 (+ 220us) client_negotiation.cc:769] Sending connection context
0929 01:06:40.064714 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 01:06:40.065028 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":29,"spinlock_wait_cycles":1280,"thread_start_us":142,"threads_started":1}
I20260929 01:06:40.065896 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:40.053341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33976 (local address 127.0.98.60:43981)
0929 01:06:40.053766 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:40.053781 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:40.053816 (+ 35us) server_negotiation.cc:407] Connection header received
0929 01:06:40.054483 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:40.054513 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:40.054945 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:40.055326 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:40.056129 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.057481 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.058333 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.058774 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.058907 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:40.061586 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:40.061591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:40.061621 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:40.061650 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:40.063467 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:40.063966 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:40.063972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:40.063973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:40.064047 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:40.064321 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:40.064324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:40.064326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:40.064569 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:40.064727 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:40.064980 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 01:06:40.065101 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":296,"thread_start_us":120,"threads_started":1}
I20260929 01:06:40.067073 22863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35615 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:06:40 dist-test-slave-7zdl krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 01:06:40 dist-test-slave-7zdl krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:06:40.074966 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:40.067561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35615 (local address 127.0.0.1:60448)
0929 01:06:40.067703 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:40.067716 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:40.067781 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:40.068037 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:40.068041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:40.068207 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:40.068320 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:40.068328 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.069140 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.069146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:40.069666 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:40.069673 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.070177 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.070181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:40.070270 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:40.070686 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:40.070696 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:40.071818 (+ 1122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:40.073513 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:40.073519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:40.073521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:40.073709 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:40.074018 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:40.074023 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:40.074025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:40.074108 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:40.074482 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:40.074485 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:40.074573 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 01:06:40.074674 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 01:06:40.074793 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 01:06:40.075049 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:40.067672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60448 (local address 127.0.98.61:35615)
0929 01:06:40.067792 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:40.067795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:40.067823 (+ 28us) server_negotiation.cc:407] Connection header received
0929 01:06:40.067865 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:40.067868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:40.067915 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:40.068002 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:40.068481 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.069026 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.069799 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.070055 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.070165 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:40.071950 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:40.071954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:40.071956 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:40.071991 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:40.073403 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:40.073806 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:40.073811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:40.073813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:40.073923 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:40.074204 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:40.074207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:40.074209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:40.074379 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:40.074480 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:40.074739 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 01:06:40.074898 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 01:06:40.075673 22863 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:06:40.076624 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:40.068274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47706 (local address 127.0.98.62:34837)
0929 01:06:40.068384 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:40.068387 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:40.068399 (+ 12us) server_negotiation.cc:407] Connection header received
0929 01:06:40.068499 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:40.068502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:40.068536 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:40.068622 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:40.069171 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.069638 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.070370 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.070727 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.070875 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:40.073278 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:40.073282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:40.073284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:40.073307 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:40.074676 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:40.075354 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:40.075358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:40.075360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:40.075439 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:40.075820 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:40.075822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:40.075823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:40.076049 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:40.076248 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:40.076402 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 01:06:40.076494 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 01:06:40 dist-test-slave-7zdl krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 01:06:40 dist-test-slave-7zdl krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 01:06:40.082372 22712 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } attempt: 3
I20260929 01:06:40.083861 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:40.076089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43981 (local address 127.0.0.1:33982)
0929 01:06:40.076230 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:40.076242 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:40.076311 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:40.076733 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:40.076737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:40.076889 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:40.076970 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:40.076976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.077893 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.077897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:40.078485 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:40.078496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.078965 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.078969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:40.079044 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:40.079482 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:40.079493 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:40.080621 (+ 1128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:40.082242 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:40.082246 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:40.082248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:40.082495 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:40.082776 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:40.082779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:40.082781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:40.082908 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:40.083361 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:40.083366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:40.083459 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 01:06:40.083561 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 01:06:40.083661 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 01:06:40.084365 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:40.076134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33982 (local address 127.0.98.60:43981)
0929 01:06:40.076475 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:40.076481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:40.076495 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:06:40.076526 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:40.076529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:40.076566 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:40.076635 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:40.077067 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.077770 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.078594 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.078861 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.078945 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:40.080754 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:40.080757 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:40.080764 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:40.080791 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:40.082138 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:40.082607 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:40.082610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:40.082612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:40.082684 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:40.083014 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:40.083019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:40.083020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:40.083232 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:40.083315 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:40.084037 (+ 722us) server_negotiation.cc:299] Negotiation successful
0929 01:06:40.084156 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280}
I20260929 01:06:40.088878 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:40.077669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60460 (local address 127.0.98.61:35615)
0929 01:06:40.077762 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:40.077765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:40.077776 (+ 11us) server_negotiation.cc:407] Connection header received
0929 01:06:40.077882 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:40.077884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:40.077924 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:40.078001 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:40.078510 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.079159 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.079892 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.080094 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.080193 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:40.082927 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:40.082934 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:40.082940 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:40.082969 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:40.086411 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:40.086943 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:40.086946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:40.086948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:40.087037 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:40.087311 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:40.087314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:40.087316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:40.087519 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:40.087699 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:40.088602 (+ 903us) server_negotiation.cc:299] Negotiation successful
0929 01:06:40.088722 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
Sep 29 01:06:40 dist-test-slave-7zdl krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 01:06:40.089507 22787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } attempt: 3
I20260929 01:06:40.089623 22863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e97fe2a6a4b4a7e9a824ad825735c43: Bootstrap starting.
I20260929 01:06:40.093086 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:40.093436 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:40.082747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43981 (local address 127.0.0.1:33996)
0929 01:06:40.083080 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:40.083094 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:40.083162 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:40.083908 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:40.083914 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:40.084086 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:40.084195 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:40.084205 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.086019 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.086025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:40.086920 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:40.086931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.087565 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.087571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:40.087650 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:40.088093 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:40.088107 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:40.089788 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:40.091811 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:40.091816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:40.091818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:40.092063 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:40.092390 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:40.092395 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:40.092400 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:40.092512 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:40.092987 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:40.092990 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:40.093085 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 01:06:40.093173 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 01:06:40.093289 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257}
I20260929 01:06:40.094084 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:40.083292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33996 (local address 127.0.98.60:43981)
0929 01:06:40.083671 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:40.083682 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:40.083697 (+ 15us) server_negotiation.cc:407] Connection header received
0929 01:06:40.083729 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:40.083731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:40.083773 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:40.084315 (+ 542us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:40.084431 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.085873 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.087054 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.087434 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.087854 (+ 420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:40.089921 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:40.089924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:40.089927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:40.089950 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:40.091702 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:40.092147 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:40.092152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:40.092154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:40.092272 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:40.092636 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:40.092640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:40.092645 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:40.092869 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:40.093720 (+ 851us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:40.093794 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 01:06:40.093903 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":116,"threads_started":1}
I20260929 01:06:40.094399 22686 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "95d4d108de8e46fea136cd61ce55b8c4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47706
I20260929 01:06:40.097434 22712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4: Bootstrap starting.
Sep 29 01:06:40 dist-test-slave-7zdl krb5kdc[22631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790643999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 01:06:40.099617 22863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e97fe2a6a4b4a7e9a824ad825735c43: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:40.100653 22712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:40.100639 22863 log.cc:824] T 00000000000000000000000000000000 P 0e97fe2a6a4b4a7e9a824ad825735c43: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:40.101511 22712 log.cc:824] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:40.102100 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:40.090276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34012 (local address 127.0.98.60:43981)
0929 01:06:40.093629 (+ 3353us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:40.093634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:40.093647 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:06:40.093678 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:40.093681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:40.093716 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:40.094213 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:40.094292 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.095216 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.096299 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.096618 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.096721 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:40.098525 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:40.098528 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:40.098531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:40.098554 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:40.100260 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:40.100667 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:40.100673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:40.100675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:40.100760 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:40.101104 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:40.101107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:40.101108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:40.101479 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:40.101564 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:40.101847 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 01:06:40.101956 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3280}
I20260929 01:06:40.102101 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:40.089827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43981 (local address 127.0.0.1:34012)
0929 01:06:40.089966 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:40.089979 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:40.090053 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:40.093842 (+ 3789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:40.093845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:40.094006 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:40.094096 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:40.094102 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.095338 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.095341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:40.096181 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:40.096194 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.096725 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.096729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:40.096814 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:40.097270 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:40.097283 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:40.098399 (+ 1116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:40.100370 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:40.100373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:40.100375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:40.100570 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:40.100881 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:40.100884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:40.100886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:40.100972 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:40.101581 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:40.101584 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:40.101679 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 01:06:40.101798 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 01:06:40.101950 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 01:06:40.102877 22863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e97fe2a6a4b4a7e9a824ad825735c43: No bootstrap required, opened a new log
I20260929 01:06:40.103473 22712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4: No bootstrap required, opened a new log
I20260929 01:06:40.104863 22787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 925e3b9e25594922aa9849d068d600e0: Bootstrap starting.
I20260929 01:06:40.106307 22712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d4d108de8e46fea136cd61ce55b8c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } } peers { permanent_uuid: "925e3b9e25594922aa9849d068d600e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35615 } } peers { permanent_uuid: "0e97fe2a6a4b4a7e9a824ad825735c43" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } }
I20260929 01:06:40.106539 22712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95d4d108de8e46fea136cd61ce55b8c4, State: Initialized, Role: FOLLOWER
I20260929 01:06:40.106786 22863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e97fe2a6a4b4a7e9a824ad825735c43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d4d108de8e46fea136cd61ce55b8c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } } peers { permanent_uuid: "925e3b9e25594922aa9849d068d600e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35615 } } peers { permanent_uuid: "0e97fe2a6a4b4a7e9a824ad825735c43" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } }
I20260929 01:06:40.106914 22712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d4d108de8e46fea136cd61ce55b8c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } } peers { permanent_uuid: "925e3b9e25594922aa9849d068d600e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35615 } } peers { permanent_uuid: "0e97fe2a6a4b4a7e9a824ad825735c43" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } }
I20260929 01:06:40.107012 22787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 925e3b9e25594922aa9849d068d600e0: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:40.107046 22863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e97fe2a6a4b4a7e9a824ad825735c43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e97fe2a6a4b4a7e9a824ad825735c43, State: Initialized, Role: FOLLOWER
I20260929 01:06:40.107334 22863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e97fe2a6a4b4a7e9a824ad825735c43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d4d108de8e46fea136cd61ce55b8c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } } peers { permanent_uuid: "925e3b9e25594922aa9849d068d600e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35615 } } peers { permanent_uuid: "0e97fe2a6a4b4a7e9a824ad825735c43" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } }
I20260929 01:06:40.107533 22871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d4d108de8e46fea136cd61ce55b8c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } } peers { permanent_uuid: "925e3b9e25594922aa9849d068d600e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35615 } } peers { permanent_uuid: "0e97fe2a6a4b4a7e9a824ad825735c43" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } } }
I20260929 01:06:40.107826 22871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:40.107834 22787 log.cc:824] T 00000000000000000000000000000000 P 925e3b9e25594922aa9849d068d600e0: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:40.108245 22712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:40.108369 22686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 01:06:40.108491 22863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e97fe2a6a4b4a7e9a824ad825735c43 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:40.108497 22686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:40.109700 22686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d4d108de8e46fea136cd61ce55b8c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } } peers { permanent_uuid: "925e3b9e25594922aa9849d068d600e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35615 } } peers { permanent_uuid: "0e97fe2a6a4b4a7e9a824ad825735c43" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } }
I20260929 01:06:40.110057 22787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 925e3b9e25594922aa9849d068d600e0: No bootstrap required, opened a new log
I20260929 01:06:40.110641 22686 leader_election.cc:288] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4 [CANDIDATE]: Term 1 election: Requested vote from peers 925e3b9e25594922aa9849d068d600e0 (127.0.98.61:35615), 0e97fe2a6a4b4a7e9a824ad825735c43 (127.0.98.60:43981)
I20260929 01:06:40.113775 22872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e97fe2a6a4b4a7e9a824ad825735c43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d4d108de8e46fea136cd61ce55b8c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } } peers { permanent_uuid: "925e3b9e25594922aa9849d068d600e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35615 } } peers { permanent_uuid: "0e97fe2a6a4b4a7e9a824ad825735c43" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } } }
I20260929 01:06:40.113914 22872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e97fe2a6a4b4a7e9a824ad825735c43 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:40.114566 22787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 925e3b9e25594922aa9849d068d600e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d4d108de8e46fea136cd61ce55b8c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } } peers { permanent_uuid: "925e3b9e25594922aa9849d068d600e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35615 } } peers { permanent_uuid: "0e97fe2a6a4b4a7e9a824ad825735c43" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } }
I20260929 01:06:40.114881 22787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 925e3b9e25594922aa9849d068d600e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 925e3b9e25594922aa9849d068d600e0, State: Initialized, Role: FOLLOWER
I20260929 01:06:40.115252 22787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 925e3b9e25594922aa9849d068d600e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d4d108de8e46fea136cd61ce55b8c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } } peers { permanent_uuid: "925e3b9e25594922aa9849d068d600e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35615 } } peers { permanent_uuid: "0e97fe2a6a4b4a7e9a824ad825735c43" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } }
I20260929 01:06:40.116062 22901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 925e3b9e25594922aa9849d068d600e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d4d108de8e46fea136cd61ce55b8c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } } peers { permanent_uuid: "925e3b9e25594922aa9849d068d600e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35615 } } peers { permanent_uuid: "0e97fe2a6a4b4a7e9a824ad825735c43" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } } }
W20260929 01:06:40.116366 22665 tablet.cc:2369] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:06:40.116832 22787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 925e3b9e25594922aa9849d068d600e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:40.116856 22761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95d4d108de8e46fea136cd61ce55b8c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "925e3b9e25594922aa9849d068d600e0"
I20260929 01:06:40.116199 22901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 925e3b9e25594922aa9849d068d600e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:40.117069 22761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 925e3b9e25594922aa9849d068d600e0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:40.118460 22761 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 925e3b9e25594922aa9849d068d600e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95d4d108de8e46fea136cd61ce55b8c4 in term 1.
I20260929 01:06:40.119087 22662 leader_election.cc:302] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 925e3b9e25594922aa9849d068d600e0, 95d4d108de8e46fea136cd61ce55b8c4; no voters:
I20260929 01:06:40.119359 22871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:40.112771 22837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95d4d108de8e46fea136cd61ce55b8c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0e97fe2a6a4b4a7e9a824ad825735c43"
I20260929 01:06:40.119567 22871 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4 [term 1 LEADER]: Becoming Leader. State: Replica: 95d4d108de8e46fea136cd61ce55b8c4, State: Running, Role: LEADER
I20260929 01:06:40.119614 22837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e97fe2a6a4b4a7e9a824ad825735c43 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:40.119887 22871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d4d108de8e46fea136cd61ce55b8c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } } peers { permanent_uuid: "925e3b9e25594922aa9849d068d600e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35615 } } peers { permanent_uuid: "0e97fe2a6a4b4a7e9a824ad825735c43" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } }
I20260929 01:06:40.121030 22837 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0e97fe2a6a4b4a7e9a824ad825735c43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95d4d108de8e46fea136cd61ce55b8c4 in term 1.
I20260929 01:06:40.123016 22871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95d4d108de8e46fea136cd61ce55b8c4. Latest consensus state: current_term: 1 leader_uuid: "95d4d108de8e46fea136cd61ce55b8c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d4d108de8e46fea136cd61ce55b8c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } } peers { permanent_uuid: "925e3b9e25594922aa9849d068d600e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35615 } } peers { permanent_uuid: "0e97fe2a6a4b4a7e9a824ad825735c43" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } } }
I20260929 01:06:40.123136 22871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:40.119665 22794 master_runner.cc:186] Error getting master registration for 127.0.98.61:35615: OK, instance_id { permanent_uuid: "925e3b9e25594922aa9849d068d600e0" instance_seqno: 1790643999830375 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:06:40.123333 22794 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:06:40.123945 22914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:40.126116 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:40.114462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34026 (local address 127.0.98.60:43981)
0929 01:06:40.114567 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:40.114572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:40.115269 (+ 697us) server_negotiation.cc:407] Connection header received
0929 01:06:40.116337 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:40.116341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:40.116393 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:40.116447 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:40.117536 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.118353 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.119590 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.119916 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.120014 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:40.121896 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:40.121900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:40.121903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:40.121933 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:40.123866 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:40.124367 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:40.124373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:40.124378 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:40.124467 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:40.124783 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:40.124789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:40.124794 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:40.125027 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:40.125148 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:40.125784 (+ 636us) server_negotiation.cc:299] Negotiation successful
0929 01:06:40.125966 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 01:06:40.128963 22914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260929 01:06:40.129281 22915 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 925e3b9e25594922aa9849d068d600e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 01:06:40.129310 22644 master_runner.cc:263] No leader master found from master 95d4d108de8e46fea136cd61ce55b8c4
I20260929 01:06:40.129381 22644 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:06:40.129972 22915 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 01:06:40.134691 22895 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0e97fe2a6a4b4a7e9a824ad825735c43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:06:40.134785 22895 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:06:40.139107 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:40.128178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60466 (local address 127.0.98.61:35615)
0929 01:06:40.128286 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:40.128290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:40.129729 (+ 1439us) server_negotiation.cc:407] Connection header received
0929 01:06:40.129767 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:40.129770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:40.129816 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:40.129897 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:40.130913 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.131646 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.132736 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.133101 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.133212 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:40.135208 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:40.135211 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:40.135214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:40.135256 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:40.137185 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:40.137600 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:40.137605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:40.137608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:40.137692 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:40.137978 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:40.137983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:40.137986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:40.138237 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:40.138421 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:40.138825 (+ 404us) server_negotiation.cc:299] Negotiation successful
0929 01:06:40.138943 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 01:06:40.149029 22761 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 925e3b9e25594922aa9849d068d600e0 [term 1 FOLLOWER]: Refusing update from remote peer 95d4d108de8e46fea136cd61ce55b8c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:06:40.150094 22871 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925e3b9e25594922aa9849d068d600e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:06:40.150815 22837 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0e97fe2a6a4b4a7e9a824ad825735c43 [term 1 FOLLOWER]: Refusing update from remote peer 95d4d108de8e46fea136cd61ce55b8c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:06:40.151441 22912 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e97fe2a6a4b4a7e9a824ad825735c43" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:06:40.158265 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:40.129790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47718 (local address 127.0.98.62:34837)
0929 01:06:40.129884 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:40.129889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:40.135539 (+ 5650us) server_negotiation.cc:407] Connection header received
0929 01:06:40.135732 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:40.135735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:40.135786 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:40.135837 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:40.145333 (+ 9496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.145827 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.146647 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:40.147037 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:40.147150 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:40.152564 (+ 5414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:40.152568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:40.152571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:40.152599 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:40.155110 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:40.155890 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:40.155897 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:40.155899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:40.156071 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:40.156522 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:40.156526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:40.156528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:40.156957 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:40.157179 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:40.157792 (+ 613us) server_negotiation.cc:299] Negotiation successful
0929 01:06:40.157994 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 01:06:40.173610 22901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 925e3b9e25594922aa9849d068d600e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95d4d108de8e46fea136cd61ce55b8c4. Latest consensus state: current_term: 1 leader_uuid: "95d4d108de8e46fea136cd61ce55b8c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d4d108de8e46fea136cd61ce55b8c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } } peers { permanent_uuid: "925e3b9e25594922aa9849d068d600e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35615 } } peers { permanent_uuid: "0e97fe2a6a4b4a7e9a824ad825735c43" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } } }
I20260929 01:06:40.173738 22901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 925e3b9e25594922aa9849d068d600e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:40.174574 22872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e97fe2a6a4b4a7e9a824ad825735c43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95d4d108de8e46fea136cd61ce55b8c4. Latest consensus state: current_term: 1 leader_uuid: "95d4d108de8e46fea136cd61ce55b8c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d4d108de8e46fea136cd61ce55b8c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } } peers { permanent_uuid: "925e3b9e25594922aa9849d068d600e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35615 } } peers { permanent_uuid: "0e97fe2a6a4b4a7e9a824ad825735c43" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } } }
I20260929 01:06:40.174706 22872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e97fe2a6a4b4a7e9a824ad825735c43 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:40.185711 22912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95d4d108de8e46fea136cd61ce55b8c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d4d108de8e46fea136cd61ce55b8c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } } peers { permanent_uuid: "925e3b9e25594922aa9849d068d600e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35615 } } peers { permanent_uuid: "0e97fe2a6a4b4a7e9a824ad825735c43" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } } }
I20260929 01:06:40.185851 22912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:40.187319 22871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95d4d108de8e46fea136cd61ce55b8c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d4d108de8e46fea136cd61ce55b8c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } } peers { permanent_uuid: "925e3b9e25594922aa9849d068d600e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35615 } } peers { permanent_uuid: "0e97fe2a6a4b4a7e9a824ad825735c43" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } } }
I20260929 01:06:40.187440 22871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4 [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:40.188494 22901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 925e3b9e25594922aa9849d068d600e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95d4d108de8e46fea136cd61ce55b8c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d4d108de8e46fea136cd61ce55b8c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } } peers { permanent_uuid: "925e3b9e25594922aa9849d068d600e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35615 } } peers { permanent_uuid: "0e97fe2a6a4b4a7e9a824ad825735c43" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } } }
I20260929 01:06:40.188619 22901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 925e3b9e25594922aa9849d068d600e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:40.188721 22921 mvcc.cc:205] Tried to move back new op lower bound from 7334477824577966080 to 7334477824497475584. Current Snapshot: MvccSnapshot[applied={T|T < 7334477824577966080}]
I20260929 01:06:40.189225 22872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e97fe2a6a4b4a7e9a824ad825735c43 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95d4d108de8e46fea136cd61ce55b8c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95d4d108de8e46fea136cd61ce55b8c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34837 } } peers { permanent_uuid: "925e3b9e25594922aa9849d068d600e0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35615 } } peers { permanent_uuid: "0e97fe2a6a4b4a7e9a824ad825735c43" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43981 } } }
I20260929 01:06:40.189347 22872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e97fe2a6a4b4a7e9a824ad825735c43 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:40.190330 22922 mvcc.cc:205] Tried to move back new op lower bound from 7334477824577966080 to 7334477824497475584. Current Snapshot: MvccSnapshot[applied={T|T < 7334477824577966080}]
I20260929 01:06:40.192077 22914 catalog_manager.cc:1347] Generated new cluster ID: 87edd83ce213473e98b58928a69ccc75
I20260929 01:06:40.192144 22914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:40.208952 22914 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:40.210237 22914 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:40.232190 22914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95d4d108de8e46fea136cd61ce55b8c4: Generated new TSK 0
I20260929 01:06:40.233122 22914 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 01:06:40.263438 22866 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:40.260572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34028 (local address 127.0.98.60:43981)
0929 01:06:40.260705 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:40.260710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:40.262334 (+ 1624us) server_negotiation.cc:407] Connection header received
0929 01:06:40.262573 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:40.262576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:40.262626 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:40.262678 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:40.263295 (+ 617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34028: BlockingRecv error: recv got EOF from 127.0.0.1:34028 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 01:06:40.264680 22792 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:40.261155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60468 (local address 127.0.98.61:35615)
0929 01:06:40.261265 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:40.261270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:40.263848 (+ 2578us) server_negotiation.cc:407] Connection header received
0929 01:06:40.264008 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:40.264011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:40.264058 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:40.264108 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:40.264567 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60468: BlockingRecv error: recv got EOF from 127.0.0.1:60468 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 01:06:40.265934 22715 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:40.261551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.0.98.62:34837)
0929 01:06:40.261661 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:40.261665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:40.265112 (+ 3447us) server_negotiation.cc:407] Connection header received
0929 01:06:40.265270 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:40.265273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:40.265319 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:40.265368 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:40.265825 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47722: BlockingRecv error: recv got EOF from 127.0.0.1:47722 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 01:06:40.269954 22792 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:40.268581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60480 (local address 127.0.98.61:35615)
0929 01:06:40.268675 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:40.268679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:40.269095 (+ 416us) server_negotiation.cc:407] Connection header received
0929 01:06:40.269290 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:40.269293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:40.269360 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:40.269406 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:40.269859 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60480: BlockingRecv error: recv got EOF from 127.0.0.1:60480 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260929 01:06:40.274636 22792 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:40.273269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60496 (local address 127.0.98.61:35615)
0929 01:06:40.273357 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:40.273361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:40.273837 (+ 476us) server_negotiation.cc:407] Connection header received
0929 01:06:40.273997 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:40.274000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:40.274044 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:40.274090 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:40.274549 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60496: BlockingRecv error: recv got EOF from 127.0.0.1:60496 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260929 01:06:40.280167 22792 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:40.278618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60502 (local address 127.0.98.61:35615)
0929 01:06:40.278706 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:40.278710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:40.279332 (+ 622us) server_negotiation.cc:407] Connection header received
0929 01:06:40.279524 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:40.279528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:40.279573 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:40.279620 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:40.280080 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60502: BlockingRecv error: recv got EOF from 127.0.0.1:60502 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260929 01:06:40.287796 22792 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:40.286237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60508 (local address 127.0.98.61:35615)
0929 01:06:40.286373 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:40.286379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:40.286913 (+ 534us) server_negotiation.cc:407] Connection header received
0929 01:06:40.287099 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:40.287102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:40.287148 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:40.287199 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:40.287691 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60508: BlockingRecv error: recv got EOF from 127.0.0.1:60508 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 01:06:40.299661 22792 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:40.297984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60514 (local address 127.0.98.61:35615)
0929 01:06:40.298092 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:40.298096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:40.298639 (+ 543us) server_negotiation.cc:407] Connection header received
0929 01:06:40.298862 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:40.298866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:40.298923 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:40.298988 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:40.299550 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60514: BlockingRecv error: recv got EOF from 127.0.0.1:60514 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 01:06:40.319591 22792 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:40.317875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60526 (local address 127.0.98.61:35615)
0929 01:06:40.318008 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:40.318012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:40.318589 (+ 577us) server_negotiation.cc:407] Connection header received
0929 01:06:40.318839 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:40.318842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:40.318892 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:40.318943 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:40.319482 (+ 539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60526: BlockingRecv error: recv got EOF from 127.0.0.1:60526 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 01:06:40.355391 22792 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:40.354046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60530 (local address 127.0.98.61:35615)
0929 01:06:40.354168 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:40.354172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:40.354404 (+ 232us) server_negotiation.cc:407] Connection header received
0929 01:06:40.354573 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:40.354576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:40.354631 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:40.354729 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:40.355267 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60530: BlockingRecv error: recv got EOF from 127.0.0.1:60530 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 01:06:40.422940 22792 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:40.421742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60544 (local address 127.0.98.61:35615)
0929 01:06:40.421942 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:40.421946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:40.421962 (+ 16us) server_negotiation.cc:407] Connection header received
0929 01:06:40.422090 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:40.422093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:40.422144 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:40.422216 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:40.422717 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60544: BlockingRecv error: recv got EOF from 127.0.0.1:60544 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
W20260929 01:06:40.553982 22792 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:40.552790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60554 (local address 127.0.98.61:35615)
0929 01:06:40.552953 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:40.552958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:40.553079 (+ 121us) server_negotiation.cc:407] Connection header received
0929 01:06:40.553249 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:40.553253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:40.553308 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:40.553398 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:40.553852 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60554: BlockingRecv error: recv got EOF from 127.0.0.1:60554 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 01:06:40.813649 22792 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:40.812467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60558 (local address 127.0.98.61:35615)
0929 01:06:40.812607 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:40.812612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:40.812682 (+ 70us) server_negotiation.cc:407] Connection header received
0929 01:06:40.812870 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:40.812873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:40.812927 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:40.813024 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:40.813505 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60558: BlockingRecv error: recv got EOF from 127.0.0.1:60558 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 01:06:41.131572 22915 catalog_manager.cc:1259] Loaded cluster ID: 87edd83ce213473e98b58928a69ccc75
I20260929 01:06:41.131695 22915 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 925e3b9e25594922aa9849d068d600e0: loading cluster ID for follower catalog manager: success
I20260929 01:06:41.133508 22915 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 925e3b9e25594922aa9849d068d600e0: acquiring CA information for follower catalog manager: success
I20260929 01:06:41.134840 22915 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 925e3b9e25594922aa9849d068d600e0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:06:41.135847 22895 catalog_manager.cc:1259] Loaded cluster ID: 87edd83ce213473e98b58928a69ccc75
I20260929 01:06:41.135905 22895 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0e97fe2a6a4b4a7e9a824ad825735c43: loading cluster ID for follower catalog manager: success
I20260929 01:06:41.137396 22895 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0e97fe2a6a4b4a7e9a824ad825735c43: acquiring CA information for follower catalog manager: success
I20260929 01:06:41.138432 22895 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0e97fe2a6a4b4a7e9a824ad825735c43: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:06:41.331888 22992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:41.327966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60570 (local address 127.0.98.61:35615)
0929 01:06:41.328197 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:41.328200 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:41.328289 (+ 89us) server_negotiation.cc:407] Connection header received
0929 01:06:41.328407 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:41.328410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:41.328537 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:41.328644 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:41.328977 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:41.329495 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:41.330405 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:41.330629 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:41.330708 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:41.331063 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:41.331488 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:41.331582 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:41.331636 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 01:06:41.331682 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":88,"threads_started":1}
I20260929 01:06:41.338325 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:41.334888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34040 (local address 127.0.98.60:43981)
0929 01:06:41.335156 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:41.335160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:41.335173 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:06:41.335309 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:41.335311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:41.335422 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:41.335514 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:41.335689 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:41.336231 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:41.336998 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:41.337218 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:41.337301 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:41.337544 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:41.337950 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:41.338035 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:41.338085 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 01:06:41.338130 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":89,"threads_started":1}
I20260929 01:06:41.345392 22992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:41.341238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60574 (local address 127.0.98.61:35615)
0929 01:06:41.341352 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:41.341357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:41.342046 (+ 689us) server_negotiation.cc:407] Connection header received
0929 01:06:41.342220 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:41.342223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:41.342276 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:41.342359 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:41.342653 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:41.343318 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:41.344126 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:41.344437 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:41.344528 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:41.344869 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:41.344943 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:41.345011 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:41.345173 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 01:06:41.345250 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 01:06:41.345463 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:41.341597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34052 (local address 127.0.98.60:43981)
0929 01:06:41.341730 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:41.341737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:41.341776 (+ 39us) server_negotiation.cc:407] Connection header received
0929 01:06:41.341859 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:41.341862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:41.341913 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:41.341990 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:41.342251 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:41.343024 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:41.344214 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:41.344541 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:41.344638 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:41.344906 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:41.344986 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:41.345049 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:41.345199 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 01:06:41.345371 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 01:06:41.346323 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:41.341262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47736 (local address 127.0.98.62:34837)
0929 01:06:41.341519 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:41.341525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:41.341754 (+ 229us) server_negotiation.cc:407] Connection header received
0929 01:06:41.341900 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:41.341905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:41.342064 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:41.342229 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:41.342636 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:41.343412 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:41.344529 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:41.344902 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:41.345010 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:41.345364 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:41.345927 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:41.346029 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:41.346110 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 01:06:41.346171 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":72,"threads_started":1}
I20260929 01:06:41.349426 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 22644
I20260929 01:06:41.356113 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 22719
I20260929 01:06:41.361908 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 22794
2026-09-29T01:06:41Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3078 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:06:41 dist-test-slave-7zdl krb5kdc[23009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:06:41 dist-test-slave-7zdl krb5kdc[23009](info): set up 2 sockets
Sep 29 01:06:41 dist-test-slave-7zdl krb5kdc[23009](info): commencing operation
krb5kdc: starting...
W20260929 01:06:42.387063 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/krb5kdc/test-user.keytab.
Sep 29 01:06:42 dist-test-slave-7zdl krb5kdc[23009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:06:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:06:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 01:06:42.510768 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34219
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34219,127.0.98.61:36709,127.0.98.60:37219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:06:42.618690 23022 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 01:06:42.619029 23022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:42.619096 23022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:42.619186 23022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:42.623169 23022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:42.623255 23022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:42.623291 23022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:42.623324 23022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:42.623360 23022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:42.628475 23022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34219,127.0.98.61:36709,127.0.98.60:37219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:42.629514 23022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:42.630523 23022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:42.639040 23028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:42.639076 23022 server_base.cc:1034] running on GCE node
W20260929 01:06:42.639132 23030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:42.639040 23027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:42.639846 23022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:42.640877 23022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:42.642061 23022 hybrid_clock.cc:648] HybridClock initialized: now 1790644002642029 us; error 45 us; skew 500 ppm
Sep 29 01:06:42 dist-test-slave-7zdl krb5kdc[23009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:42.650028 23022 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 01:06:42.650972 23022 webserver.cc:460] Webserver started at http://127.0.98.62:38661/ using document root <none> and password file <none>
I20260929 01:06:42.651520 23022 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:42.651608 23022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:42.651829 23022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:42.653789 23022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-0/data/instance:
uuid: "df1b6bfb352341c8b9dbef768d7919eb"
format_stamp: "Formatted at 2026-09-29 01:06:42 on dist-test-slave-7zdl"
server_key: "5f9e926722681156b06dd41778dc2b03"
server_key_iv: "02be93933685fb92b9855bb002303be4"
server_key_version: "encryptionkey@0"
I20260929 01:06:42.654413 23022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-0/wal/instance:
uuid: "df1b6bfb352341c8b9dbef768d7919eb"
format_stamp: "Formatted at 2026-09-29 01:06:42 on dist-test-slave-7zdl"
server_key: "5f9e926722681156b06dd41778dc2b03"
server_key_iv: "02be93933685fb92b9855bb002303be4"
server_key_version: "encryptionkey@0"
I20260929 01:06:42.658247 23022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 01:06:42.660689 23037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:42.661867 23022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:06:42.662004 23022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-0/wal
uuid: "df1b6bfb352341c8b9dbef768d7919eb"
format_stamp: "Formatted at 2026-09-29 01:06:42 on dist-test-slave-7zdl"
server_key: "5f9e926722681156b06dd41778dc2b03"
server_key_iv: "02be93933685fb92b9855bb002303be4"
server_key_version: "encryptionkey@0"
I20260929 01:06:42.662111 23022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:42.705632 23022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:42.709141 23022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:42.709353 23022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:42.716279 23022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34219
I20260929 01:06:42.716595 23089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34219 every 8 connection(s)
I20260929 01:06:42.717358 23022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-0/data/info.pb
I20260929 01:06:42.720166 23090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:42.723834 23090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34219 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:06:42.727049 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 23022
I20260929 01:06:42.727180 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-0/wal/instance
I20260929 01:06:42.727458 392 external_mini_cluster.cc:1428] Setting key 75b4b84d08423b7c9a47fe3d52f60129
Sep 29 01:06:42 dist-test-slave-7zdl krb5kdc[23009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260929 01:06:42.739044 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.726070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51328 (local address 127.0.98.62:34219)
0929 01:06:42.726592 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:42.726610 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:42.726897 (+ 287us) server_negotiation.cc:407] Connection header received
0929 01:06:42.728060 (+ 1163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:42.728071 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:42.728287 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:42.728488 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:42.729334 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.730153 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.731238 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.731600 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.731777 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:42.734374 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:42.734380 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:42.734389 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:42.734413 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:42.736037 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:42.736508 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:42.736516 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:42.736522 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:42.736618 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:42.736930 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:42.736935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:42.736937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:42.737304 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:42.737448 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:42.737901 (+ 453us) server_negotiation.cc:299] Negotiation successful
0929 01:06:42.738194 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":344,"spinlock_wait_cycles":73600,"thread_start_us":149,"threads_started":1}
I20260929 01:06:42.739049 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.726071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34219 (local address 127.0.0.1:51328)
0929 01:06:42.726679 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.726717 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:42.727743 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:42.728482 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:42.728487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:42.728946 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:42.729170 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:42.729183 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.730291 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.730299 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:42.731122 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:42.731131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.731752 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.731765 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:42.731874 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:42.732512 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:42.732528 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:42.734212 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:42.736145 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:42.736155 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:42.736165 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:42.736397 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:42.736712 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:42.736717 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:42.736719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:42.736833 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:42.737419 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:42.737425 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:42.737564 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 01:06:42.737739 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 01:06:42.738079 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":138,"threads_started":1}
I20260929 01:06:42.741587 23090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36709 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:06:42.742292 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.742050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36709 (local address 127.0.0.1:32904)
0929 01:06:42.742163 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.742235 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 01:06:42.742609 23039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36709: connect: Connection refused (error 111)
I20260929 01:06:42.743462 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.743252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36709 (local address 127.0.0.1:32906)
0929 01:06:42.743365 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.743414 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 01:06:42.743623 23090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36709: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36709: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 01:06:42.754444 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36709
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34219,127.0.98.61:36709,127.0.98.60:37219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:06:42.792840 23090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36709 } attempt: 1
I20260929 01:06:42.793812 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.793459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36709 (local address 127.0.0.1:32920)
0929 01:06:42.793637 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.793713 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 01:06:42.794580 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.794333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36709 (local address 127.0.0.1:32926)
0929 01:06:42.794453 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.794509 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 01:06:42.794880 23090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36709: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36709: connect: Connection refused (error 111)
I20260929 01:06:42.863071 23090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36709 } attempt: 2
I20260929 01:06:42.863929 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.863573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36709 (local address 127.0.0.1:32936)
0929 01:06:42.863751 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.863831 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 01:06:42.863783 23097 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 01:06:42.864084 23097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:42.864151 23097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:42.864200 23097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 01:06:42.864669 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.864428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36709 (local address 127.0.0.1:32950)
0929 01:06:42.864536 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.864587 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 01:06:42.864790 23090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36709: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36709: connect: Connection refused (error 111)
W20260929 01:06:42.868372 23097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:42.868439 23097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:42.868484 23097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:42.868520 23097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:42.868553 23097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:42.873692 23097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34219,127.0.98.61:36709,127.0.98.60:37219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:42.874715 23097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:42.875696 23097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 01:06:42.884076 23105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:42.884078 23103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:42.884094 23102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:42.884346 23097 server_base.cc:1034] running on GCE node
I20260929 01:06:42.884980 23097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:42.886140 23097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:42.887326 23097 hybrid_clock.cc:648] HybridClock initialized: now 1790644002887308 us; error 31 us; skew 500 ppm
Sep 29 01:06:42 dist-test-slave-7zdl krb5kdc[23009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:42.890077 23097 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 01:06:42.891050 23097 webserver.cc:460] Webserver started at http://127.0.98.61:35767/ using document root <none> and password file <none>
I20260929 01:06:42.891620 23097 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:42.891713 23097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:42.891956 23097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:42.894021 23097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-1/data/instance:
uuid: "7ee7a743fcb249a79ac3a7edeb8055c7"
format_stamp: "Formatted at 2026-09-29 01:06:42 on dist-test-slave-7zdl"
server_key: "82c202d8560955b2337ac5c0cc689f6d"
server_key_iv: "19c3cccca4b27a17171f048085ab6305"
server_key_version: "encryptionkey@0"
I20260929 01:06:42.894574 23097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-1/wal/instance:
uuid: "7ee7a743fcb249a79ac3a7edeb8055c7"
format_stamp: "Formatted at 2026-09-29 01:06:42 on dist-test-slave-7zdl"
server_key: "82c202d8560955b2337ac5c0cc689f6d"
server_key_iv: "19c3cccca4b27a17171f048085ab6305"
server_key_version: "encryptionkey@0"
I20260929 01:06:42.898263 23097 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:06:42.900760 23112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:42.901954 23097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:06:42.902088 23097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-1/wal
uuid: "7ee7a743fcb249a79ac3a7edeb8055c7"
format_stamp: "Formatted at 2026-09-29 01:06:42 on dist-test-slave-7zdl"
server_key: "82c202d8560955b2337ac5c0cc689f6d"
server_key_iv: "19c3cccca4b27a17171f048085ab6305"
server_key_version: "encryptionkey@0"
I20260929 01:06:42.902196 23097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:42.913537 23097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:42.916400 23097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:42.916596 23097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:42.923352 23097 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36709
I20260929 01:06:42.923610 23164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36709 every 8 connection(s)
I20260929 01:06:42.924301 23097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-1/data/info.pb
I20260929 01:06:42.927031 23165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:42.930693 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 23097
I20260929 01:06:42.930841 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-1/wal/instance
I20260929 01:06:42.930579 23165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34219 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:06:42.931151 392 external_mini_cluster.cc:1428] Setting key a8e828f27c237f981950efeae642b547
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 29 01:06:42 dist-test-slave-7zdl krb5kdc[23009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 01:06:42.946192 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.932761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51344 (local address 127.0.98.62:34219)
0929 01:06:42.932888 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:42.932892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:42.933883 (+ 991us) server_negotiation.cc:407] Connection header received
0929 01:06:42.934813 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:42.934818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:42.934886 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:42.934992 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:42.936097 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.936631 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.937777 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.938031 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.938148 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:42.942504 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:42.942507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:42.942510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:42.942535 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:42.944048 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:42.944569 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:42.944573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:42.944575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:42.944655 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:42.944982 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:42.944985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:42.944987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:42.945279 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:42.945387 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:42.945882 (+ 495us) server_negotiation.cc:299] Negotiation successful
0929 01:06:42.946021 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 01:06:42.947011 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.933149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34219 (local address 127.0.0.1:51344)
0929 01:06:42.933668 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.933745 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:42.934577 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:42.935159 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:42.935166 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:42.935504 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:42.935952 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:42.935964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.936762 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.936765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:42.937643 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:42.937652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.938183 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.938187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:42.938359 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:42.939021 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:42.939036 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:42.942302 (+ 3266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:42.944165 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:42.944171 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:42.944186 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:42.944468 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:42.944751 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:42.944754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:42.944756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:42.944886 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:42.945383 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:42.945389 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:42.945645 (+ 256us) client_negotiation.cc:769] Sending connection context
0929 01:06:42.945828 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 01:06:42.946143 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":106,"threads_started":1}
I20260929 01:06:42.948511 23165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36709 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:06:42 dist-test-slave-7zdl krb5kdc[23009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 01:06:42.955932 23090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36709 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 01:06:42.957566 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.949018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36709 (local address 127.0.0.1:32952)
0929 01:06:42.949191 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.949204 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:42.949283 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:42.949735 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:42.949737 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:42.949863 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:42.949950 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:42.949955 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.950904 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.950908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:42.951580 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:42.951591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.952164 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.952167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:42.952260 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:42.952756 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:42.952768 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:42.954007 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:42.955696 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:42.955698 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:42.955700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:42.955931 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:42.956315 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:42.956321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:42.956324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:42.956433 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:42.957025 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:42.957030 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:42.957117 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 01:06:42.957227 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 01:06:42.957372 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 01:06:42.957870 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.949108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32952 (local address 127.0.98.61:36709)
0929 01:06:42.949409 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:42.949418 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:42.949436 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:06:42.949469 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:42.949473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:42.949612 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:42.949712 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:42.950043 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.950736 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.951704 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.952063 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.952145 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:42.954136 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:42.954143 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:42.954154 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:42.954174 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:42.955616 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:42.956095 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:42.956100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:42.956102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:42.956204 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:42.956556 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:42.956558 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:42.956560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:42.956858 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:42.957008 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:42.957353 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 01:06:42.957516 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":128,"threads_started":1}
I20260929 01:06:42.958104 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
/tmp/dist-test-taskS2TZOM/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37219
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34219,127.0.98.61:36709,127.0.98.60:37219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:06:42.959911 23165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:06:42.960837 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.960344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37219 (local address 127.0.0.1:53082)
0929 01:06:42.960687 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.960767 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 01:06:42.961344 23116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37219: connect: Connection refused (error 111)
Sep 29 01:06:42 dist-test-slave-7zdl krb5kdc[23009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 01:06:42.962219 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.961916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37219 (local address 127.0.0.1:53086)
0929 01:06:42.962136 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.962175 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260929 01:06:42.962426 23165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37219: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37219: connect: Connection refused (error 111)
I20260929 01:06:42.965734 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.956448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.0.98.61:36709)
0929 01:06:42.956764 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:42.956771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:42.956782 (+ 11us) server_negotiation.cc:407] Connection header received
0929 01:06:42.956811 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:42.956814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:42.956846 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:42.956922 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:42.957418 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.958181 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.959593 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.960006 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.960501 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:42.962446 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:42.962465 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:42.962469 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:42.962491 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:42.963983 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:42.964470 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:42.964473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:42.964474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:42.964541 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:42.964855 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:42.964857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:42.964858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:42.965078 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:42.965168 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:42.965470 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 01:06:42.965553 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":164,"threads_started":1}
I20260929 01:06:42.965803 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.956414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36709 (local address 127.0.0.1:32962)
0929 01:06:42.956611 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.956627 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:42.956700 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:42.956978 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:42.956981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:42.957182 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:42.957295 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:42.957302 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.958330 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.958336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:42.959442 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:42.959455 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:42.960140 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:42.960146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:42.960273 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:42.960707 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:42.960725 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:42.962302 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:42.964093 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:42.964098 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:42.964102 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:42.964374 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:42.964633 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:42.964639 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:42.964643 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:42.964744 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:42.965168 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:42.965174 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:42.965298 (+ 124us) client_negotiation.cc:769] Sending connection context
0929 01:06:42.965417 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 01:06:42.965574 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 01:06:42.966409 23090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:06:42.967077 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.966879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37219 (local address 127.0.0.1:53088)
0929 01:06:42.966979 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.967023 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 01:06:42.967746 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.967536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37219 (local address 127.0.0.1:53094)
0929 01:06:42.967667 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.967706 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260929 01:06:42.967929 23090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37219: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37219: connect: Connection refused (error 111)
I20260929 01:06:42.999090 23090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } attempt: 1
I20260929 01:06:42.999960 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:42.999631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37219 (local address 127.0.0.1:53110)
0929 01:06:42.999828 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:42.999885 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 01:06:43.000810 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.000571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37219 (local address 127.0.0.1:53118)
0929 01:06:43.000674 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:43.000717 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 01:06:43.000996 23090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37219: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37219: connect: Connection refused (error 111)
I20260929 01:06:43.011682 23165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } attempt: 1
I20260929 01:06:43.012442 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.012156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37219 (local address 127.0.0.1:53132)
0929 01:06:43.012300 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:43.012373 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 01:06:43.013167 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.012953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37219 (local address 127.0.0.1:53144)
0929 01:06:43.013064 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:43.013113 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 01:06:43.013309 23165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37219: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37219: connect: Connection refused (error 111)
W20260929 01:06:43.069029 23172 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 01:06:43.069320 23172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:06:43.069367 23172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:06:43.069442 23172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 01:06:43.073287 23172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:06:43.073361 23172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 01:06:43.073405 23172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:06:43.073441 23172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:06:43.073474 23172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:06:43.076176 23090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } attempt: 2
I20260929 01:06:43.076997 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.076685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37219 (local address 127.0.0.1:53156)
0929 01:06:43.076840 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:43.076903 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 01:06:43.077678 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.077479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37219 (local address 127.0.0.1:53168)
0929 01:06:43.077583 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:43.077632 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 01:06:43.077898 23090 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37219: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37219: connect: Connection refused (error 111)
I20260929 01:06:43.078877 23172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34219,127.0.98.61:36709,127.0.98.60:37219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 00:14:53 UTC on ccycloud.quasar-tkanyt.root.comops.site
build id 4679058
I20260929 01:06:43.079877 23172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:06:43.080825 23172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 01:06:43.081456 23165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } attempt: 2
I20260929 01:06:43.082089 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.081846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37219 (local address 127.0.0.1:53180)
0929 01:06:43.081983 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:43.082030 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 01:06:43.082785 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.082575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37219 (local address 127.0.0.1:53182)
0929 01:06:43.082679 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:43.082734 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 01:06:43.083001 23165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37219: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37219: connect: Connection refused (error 111)
W20260929 01:06:43.089347 23178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:43.089468 23172 server_base.cc:1034] running on GCE node
W20260929 01:06:43.089324 23179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:06:43.089350 23181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:06:43.090075 23172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:06:43.091210 23172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:06:43.092393 23172 hybrid_clock.cc:648] HybridClock initialized: now 1790644003092382 us; error 22 us; skew 500 ppm
Sep 29 01:06:43 dist-test-slave-7zdl krb5kdc[23009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:06:43.095047 23172 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 01:06:43.095873 23172 webserver.cc:460] Webserver started at http://127.0.98.60:40771/ using document root <none> and password file <none>
I20260929 01:06:43.096402 23172 fs_manager.cc:359] Metadata directory not provided
I20260929 01:06:43.096488 23172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:06:43.096700 23172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:06:43.098644 23172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-2/data/instance:
uuid: "f759fb68bdac4b619ad7874d0d839b22"
format_stamp: "Formatted at 2026-09-29 01:06:43 on dist-test-slave-7zdl"
server_key: "bb479d3f469fcb2b70dfd86ea235cf99"
server_key_iv: "bc2b3eff12e76dd81abf0cb7531deacf"
server_key_version: "encryptionkey@0"
I20260929 01:06:43.099247 23172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-2/wal/instance:
uuid: "f759fb68bdac4b619ad7874d0d839b22"
format_stamp: "Formatted at 2026-09-29 01:06:43 on dist-test-slave-7zdl"
server_key: "bb479d3f469fcb2b70dfd86ea235cf99"
server_key_iv: "bc2b3eff12e76dd81abf0cb7531deacf"
server_key_version: "encryptionkey@0"
I20260929 01:06:43.102619 23172 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:06:43.105049 23188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:06:43.106410 23172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:06:43.106550 23172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-2/wal
uuid: "f759fb68bdac4b619ad7874d0d839b22"
format_stamp: "Formatted at 2026-09-29 01:06:43 on dist-test-slave-7zdl"
server_key: "bb479d3f469fcb2b70dfd86ea235cf99"
server_key_iv: "bc2b3eff12e76dd81abf0cb7531deacf"
server_key_version: "encryptionkey@0"
I20260929 01:06:43.106652 23172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:06:43.139238 23172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 01:06:43.142236 23172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:06:43.142438 23172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:06:43.149154 23172 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37219
I20260929 01:06:43.149456 23240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37219 every 8 connection(s)
I20260929 01:06:43.150104 23172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-2/data/info.pb
I20260929 01:06:43.152792 23241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:06:43.154834 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu as pid 23172
I20260929 01:06:43.154968 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskS2TZOM/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790643720886468-392-0/minicluster-data/master-2/wal/instance
I20260929 01:06:43.155259 392 external_mini_cluster.cc:1428] Setting key 916db7156cb5e1015af5f244881fe5b3
I20260929 01:06:43.157132 23241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34219 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:06:43 dist-test-slave-7zdl krb5kdc[23009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 01:06:43 dist-test-slave-7zdl krb5kdc[23009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 01:06:43.169062 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.159401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51348 (local address 127.0.98.62:34219)
0929 01:06:43.159560 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.159568 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.159814 (+ 246us) server_negotiation.cc:407] Connection header received
0929 01:06:43.159944 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.159947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.160014 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.160097 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.160778 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.161276 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.162175 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.162392 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.162524 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.165044 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:43.165050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:43.165052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:43.165076 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:43.166858 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.167377 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.167382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.167385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.167482 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.167860 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.167862 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.167864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.168100 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:43.168197 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.168771 (+ 574us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.168893 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 01:06:43.170008 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.159343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34219 (local address 127.0.0.1:51348)
0929 01:06:43.159689 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:43.159733 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:43.159843 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:43.160145 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:43.160149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:43.160526 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:43.160644 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:43.160650 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.161397 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.161400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:43.162045 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:43.162054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.162557 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.162561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:43.162664 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.163175 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:43.163190 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:43.164883 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:43.166972 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:43.166979 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:43.166986 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:43.167276 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:43.167581 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:43.167587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:43.167591 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:43.167756 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:43.168211 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:43.168216 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:43.168516 (+ 300us) client_negotiation.cc:769] Sending connection context
0929 01:06:43.168701 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 01:06:43.169018 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":27,"spinlock_wait_cycles":8448,"thread_start_us":132,"threads_started":1}
I20260929 01:06:43.170008 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.156595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53186 (local address 127.0.98.60:37219)
0929 01:06:43.157003 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.157018 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.157053 (+ 35us) server_negotiation.cc:407] Connection header received
0929 01:06:43.157823 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.157841 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.158220 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.158497 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.159359 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.160642 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.161496 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.162012 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.162178 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.165510 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:43.165516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:43.165553 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:43.165580 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:43.167426 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.167939 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.167943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.167944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.168010 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.168311 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.168334 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.168335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.168599 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:43.168780 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.169009 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.169120 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":257,"spinlock_wait_cycles":2432,"thread_start_us":131,"threads_started":1}
I20260929 01:06:43.171368 23241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36709 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:06:43.174158 23165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } attempt: 3
Sep 29 01:06:43 dist-test-slave-7zdl krb5kdc[23009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 01:06:43.177050 23090 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } attempt: 3
Sep 29 01:06:43 dist-test-slave-7zdl krb5kdc[23009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 01:06:43 dist-test-slave-7zdl krb5kdc[23009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 01:06:43.180414 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.172087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32978 (local address 127.0.98.61:36709)
0929 01:06:43.172192 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.172196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.172210 (+ 14us) server_negotiation.cc:407] Connection header received
0929 01:06:43.172252 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.172254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.172289 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.172380 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.172791 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.173308 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.174036 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.174379 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.174497 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.176680 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:43.176684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:43.176688 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:43.176720 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:43.178515 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.179037 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.179041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.179042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.179131 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.179438 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.179442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.179443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.179647 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:43.179704 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.180085 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.180235 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 01:06:43.180905 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.171802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36709 (local address 127.0.0.1:32978)
0929 01:06:43.171926 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:43.171942 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:43.172015 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:43.172392 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:43.172394 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:43.172556 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:43.172662 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:43.172673 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.173415 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.173422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:43.173918 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:43.173925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.174496 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.174500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:43.174594 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.175028 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:43.175042 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:43.176472 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:43.178636 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:43.178639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:43.178641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:43.178916 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:43.179242 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:43.179244 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:43.179246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:43.179339 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:43.179761 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:43.179763 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:43.179873 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 01:06:43.180593 (+ 720us) client_negotiation.cc:240] Negotiation successful
0929 01:06:43.180750 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 01:06:43.181562 23241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:06:43.182348 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.172541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51360 (local address 127.0.98.62:34219)
0929 01:06:43.172648 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.172651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.172660 (+ 9us) server_negotiation.cc:407] Connection header received
0929 01:06:43.172686 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.172687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.172717 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.172802 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.173348 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.173811 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.174727 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.175087 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.175169 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.177920 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:43.177923 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:43.177926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:43.177952 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:43.180055 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.180539 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.180544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.180547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.180627 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.181023 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.181028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.181031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.181459 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:43.181661 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.182055 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.182169 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 01:06:43.183892 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.174500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37219 (local address 127.0.0.1:53194)
0929 01:06:43.174644 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:43.174659 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:43.175009 (+ 350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:43.175265 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:43.175268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:43.175436 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:43.175503 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:43.175507 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.176565 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.176571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:43.177433 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:43.177444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.178098 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.178102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:43.178205 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.178611 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:43.178629 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:43.180082 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:43.182283 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:43.182287 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:43.182289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:43.182573 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:43.182911 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:43.182915 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:43.182917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:43.183009 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:43.183452 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:43.183455 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:43.183547 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 01:06:43.183646 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 01:06:43.183783 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 01:06:43.184360 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.174711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53194 (local address 127.0.98.60:37219)
0929 01:06:43.174819 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.174823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.174835 (+ 12us) server_negotiation.cc:407] Connection header received
0929 01:06:43.175134 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.175136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.175170 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.175251 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.175621 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.176438 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.177620 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.177969 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.178415 (+ 446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.180219 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:43.180224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:43.180226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:43.180251 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:43.182176 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.182671 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.182677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.182681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.182763 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.183115 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.183119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.183120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.183305 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:43.183403 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.184107 (+ 704us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.184212 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Sep 29 01:06:43 dist-test-slave-7zdl krb5kdc[23009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 01:06:43.187626 23165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ee7a743fcb249a79ac3a7edeb8055c7: Bootstrap starting.
Sep 29 01:06:43 dist-test-slave-7zdl krb5kdc[23009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 01:06:43.189549 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.177389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37219 (local address 127.0.0.1:53196)
0929 01:06:43.177886 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:43.177901 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:43.177981 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:43.179428 (+ 1447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:43.179430 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:43.179613 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:43.179707 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:43.179717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.181703 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.181707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:43.182272 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:43.182281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.183370 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.183374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:43.183455 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.184114 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:43.184127 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:43.185451 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:43.187453 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:43.187458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:43.187461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:43.187701 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:43.188489 (+ 788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:43.188493 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:43.188495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:43.188583 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:43.189056 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:43.189062 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:43.189158 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 01:06:43.189263 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 01:06:43.189398 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406}
Sep 29 01:06:43 dist-test-slave-7zdl krb5kdc[23009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790644002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 01:06:43.190109 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.178128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53196 (local address 127.0.98.60:37219)
0929 01:06:43.179183 (+ 1055us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.179198 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.179211 (+ 13us) server_negotiation.cc:407] Connection header received
0929 01:06:43.179241 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.179244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.179281 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.179489 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.179825 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.181580 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.182943 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.183259 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.183372 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.185586 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:43.185589 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:43.185592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:43.185614 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:43.187341 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.187827 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.187830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.187832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.187925 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.188701 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.188706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.188710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.188922 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:43.189704 (+ 782us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.189777 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.189903 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":985,"spinlock_wait_cycles":2304,"thread_start_us":380,"threads_started":1}
I20260929 01:06:43.190961 23165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ee7a743fcb249a79ac3a7edeb8055c7: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:43.191855 23165 log.cc:824] T 00000000000000000000000000000000 P 7ee7a743fcb249a79ac3a7edeb8055c7: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:43.193140 23090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb: Bootstrap starting.
I20260929 01:06:43.193953 23165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ee7a743fcb249a79ac3a7edeb8055c7: No bootstrap required, opened a new log
I20260929 01:06:43.194265 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.182536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37219 (local address 127.0.0.1:53200)
0929 01:06:43.183503 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0929 01:06:43.183518 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 01:06:43.183880 (+ 362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 01:06:43.184825 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 01:06:43.184829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 01:06:43.184989 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 01:06:43.185062 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:43.185067 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.186092 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.186098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:43.186977 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 01:06:43.186987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.187659 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.187664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 01:06:43.187811 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.188239 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 01:06:43.188254 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 01:06:43.190020 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 01:06:43.192665 (+ 2645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:43.192667 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:43.192669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:43.192907 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:43.193184 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 01:06:43.193189 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 01:06:43.193192 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 01:06:43.193354 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 01:06:43.193732 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 01:06:43.193734 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 01:06:43.193814 (+ 80us) client_negotiation.cc:769] Sending connection context
0929 01:06:43.193916 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 01:06:43.194114 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"spinlock_wait_cycles":1280}
I20260929 01:06:43.194445 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.184200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32980 (local address 127.0.98.61:36709)
0929 01:06:43.184297 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.184301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.184677 (+ 376us) server_negotiation.cc:407] Connection header received
0929 01:06:43.184839 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.184842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.184884 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.184943 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.185762 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.186399 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.187433 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.187726 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.187795 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.190489 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:43.190493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:43.190497 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:43.190522 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:43.192538 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.193022 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.193026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.193027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.193095 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.193404 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.193411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.193415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.193608 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:43.193800 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.194172 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.194274 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 01:06:43.195709 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.182702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.0.98.60:37219)
0929 01:06:43.183030 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.183040 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.183846 (+ 806us) server_negotiation.cc:407] Connection header received
0929 01:06:43.184357 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.184360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.184610 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.184703 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.185221 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.185966 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.187106 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.187480 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.187565 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.190332 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:43.190337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:43.190339 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:43.190359 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:43.192567 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.193012 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.193014 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.193016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.193089 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.193449 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.193454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.193458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.193619 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:43.193698 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.194910 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.195018 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":133,"threads_started":1}
I20260929 01:06:43.195891 23090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:43.197151 23090 log.cc:824] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:43.197959 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 01:06:43.198141 23165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ee7a743fcb249a79ac3a7edeb8055c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1b6bfb352341c8b9dbef768d7919eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34219 } } peers { permanent_uuid: "7ee7a743fcb249a79ac3a7edeb8055c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36709 } } peers { permanent_uuid: "f759fb68bdac4b619ad7874d0d839b22" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } }
I20260929 01:06:43.198338 23165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ee7a743fcb249a79ac3a7edeb8055c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ee7a743fcb249a79ac3a7edeb8055c7, State: Initialized, Role: FOLLOWER
I20260929 01:06:43.198599 23241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f759fb68bdac4b619ad7874d0d839b22: Bootstrap starting.
I20260929 01:06:43.198649 23165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ee7a743fcb249a79ac3a7edeb8055c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1b6bfb352341c8b9dbef768d7919eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34219 } } peers { permanent_uuid: "7ee7a743fcb249a79ac3a7edeb8055c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36709 } } peers { permanent_uuid: "f759fb68bdac4b619ad7874d0d839b22" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } }
I20260929 01:06:43.199144 23090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb: No bootstrap required, opened a new log
I20260929 01:06:43.199945 23165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ee7a743fcb249a79ac3a7edeb8055c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:43.199293 23064 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "df1b6bfb352341c8b9dbef768d7919eb"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51360
I20260929 01:06:43.201076 23241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f759fb68bdac4b619ad7874d0d839b22: Neither blocks nor log segments found. Creating new log.
I20260929 01:06:43.201838 23241 log.cc:824] T 00000000000000000000000000000000 P f759fb68bdac4b619ad7874d0d839b22: Log is configured to *not* fsync() on all Append() calls
I20260929 01:06:43.202540 23250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ee7a743fcb249a79ac3a7edeb8055c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1b6bfb352341c8b9dbef768d7919eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34219 } } peers { permanent_uuid: "7ee7a743fcb249a79ac3a7edeb8055c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36709 } } peers { permanent_uuid: "f759fb68bdac4b619ad7874d0d839b22" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } } }
I20260929 01:06:43.202663 23250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ee7a743fcb249a79ac3a7edeb8055c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:43.203118 23090 raft_consensus.cc:348] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1b6bfb352341c8b9dbef768d7919eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34219 } } peers { permanent_uuid: "7ee7a743fcb249a79ac3a7edeb8055c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36709 } } peers { permanent_uuid: "f759fb68bdac4b619ad7874d0d839b22" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } }
I20260929 01:06:43.203320 23090 raft_consensus.cc:729] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1b6bfb352341c8b9dbef768d7919eb, State: Initialized, Role: FOLLOWER
I20260929 01:06:43.203612 23090 consensus_queue.cc:260] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1b6bfb352341c8b9dbef768d7919eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34219 } } peers { permanent_uuid: "7ee7a743fcb249a79ac3a7edeb8055c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36709 } } peers { permanent_uuid: "f759fb68bdac4b619ad7874d0d839b22" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } }
I20260929 01:06:43.204181 23263 sys_catalog.cc:455] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1b6bfb352341c8b9dbef768d7919eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34219 } } peers { permanent_uuid: "7ee7a743fcb249a79ac3a7edeb8055c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36709 } } peers { permanent_uuid: "f759fb68bdac4b619ad7874d0d839b22" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } } }
I20260929 01:06:43.204448 23263 sys_catalog.cc:458] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:43.204910 23090 sys_catalog.cc:564] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:43.204988 23064 raft_consensus.cc:482] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 01:06:43.205072 23064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb [term 0 FOLLOWER]: Advancing to term 1
W20260929 01:06:43.206058 23261 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ee7a743fcb249a79ac3a7edeb8055c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:06:43.206118 23261 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:06:43.206326 23064 raft_consensus.cc:504] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1b6bfb352341c8b9dbef768d7919eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34219 } } peers { permanent_uuid: "7ee7a743fcb249a79ac3a7edeb8055c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36709 } } peers { permanent_uuid: "f759fb68bdac4b619ad7874d0d839b22" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } }
I20260929 01:06:43.203615 23097 master_runner.cc:186] Error getting master registration for 127.0.98.62:34219: OK, instance_id { permanent_uuid: "df1b6bfb352341c8b9dbef768d7919eb" instance_seqno: 1790644002714301 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:06:43.206702 23097 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:06:43.207727 23139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df1b6bfb352341c8b9dbef768d7919eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7ee7a743fcb249a79ac3a7edeb8055c7"
I20260929 01:06:43.207967 23139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ee7a743fcb249a79ac3a7edeb8055c7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:43.209277 23139 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7ee7a743fcb249a79ac3a7edeb8055c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1b6bfb352341c8b9dbef768d7919eb in term 1.
I20260929 01:06:43.207834 23064 leader_election.cc:288] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb [CANDIDATE]: Term 1 election: Requested vote from peers 7ee7a743fcb249a79ac3a7edeb8055c7 (127.0.98.61:36709), f759fb68bdac4b619ad7874d0d839b22 (127.0.98.60:37219)
I20260929 01:06:43.209663 23039 leader_election.cc:302] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ee7a743fcb249a79ac3a7edeb8055c7, df1b6bfb352341c8b9dbef768d7919eb; no voters:
I20260929 01:06:43.210027 23263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:06:43.210270 23263 raft_consensus.cc:686] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb [term 1 LEADER]: Becoming Leader. State: Replica: df1b6bfb352341c8b9dbef768d7919eb, State: Running, Role: LEADER
I20260929 01:06:43.210355 23241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f759fb68bdac4b619ad7874d0d839b22: No bootstrap required, opened a new log
I20260929 01:06:43.210776 23263 consensus_queue.cc:237] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1b6bfb352341c8b9dbef768d7919eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34219 } } peers { permanent_uuid: "7ee7a743fcb249a79ac3a7edeb8055c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36709 } } peers { permanent_uuid: "f759fb68bdac4b619ad7874d0d839b22" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } }
W20260929 01:06:43.211999 23043 tablet.cc:2369] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:06:43.212687 23263 sys_catalog.cc:455] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader df1b6bfb352341c8b9dbef768d7919eb. Latest consensus state: current_term: 1 leader_uuid: "df1b6bfb352341c8b9dbef768d7919eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1b6bfb352341c8b9dbef768d7919eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34219 } } peers { permanent_uuid: "7ee7a743fcb249a79ac3a7edeb8055c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36709 } } peers { permanent_uuid: "f759fb68bdac4b619ad7874d0d839b22" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } } }
I20260929 01:06:43.212852 23263 sys_catalog.cc:458] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb [sys.catalog]: This master's current role is: LEADER
W20260929 01:06:43.213905 23275 catalog_manager.cc:1558] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:06:43.213979 23275 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:06:43.214048 23284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 01:06:43.214993 23284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 01:06:43.214447 23022 master_runner.cc:186] Error getting master registration for 127.0.98.60:37219: OK, instance_id { permanent_uuid: "f759fb68bdac4b619ad7874d0d839b22" instance_seqno: 1790644003147273 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:06:43.218132 23022 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:06:43.220346 23241 raft_consensus.cc:348] T 00000000000000000000000000000000 P f759fb68bdac4b619ad7874d0d839b22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1b6bfb352341c8b9dbef768d7919eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34219 } } peers { permanent_uuid: "7ee7a743fcb249a79ac3a7edeb8055c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36709 } } peers { permanent_uuid: "f759fb68bdac4b619ad7874d0d839b22" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } }
I20260929 01:06:43.220582 23241 raft_consensus.cc:729] T 00000000000000000000000000000000 P f759fb68bdac4b619ad7874d0d839b22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f759fb68bdac4b619ad7874d0d839b22, State: Initialized, Role: FOLLOWER
I20260929 01:06:43.220886 23241 consensus_queue.cc:260] T 00000000000000000000000000000000 P f759fb68bdac4b619ad7874d0d839b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1b6bfb352341c8b9dbef768d7919eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34219 } } peers { permanent_uuid: "7ee7a743fcb249a79ac3a7edeb8055c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36709 } } peers { permanent_uuid: "f759fb68bdac4b619ad7874d0d839b22" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } }
I20260929 01:06:43.222167 23241 sys_catalog.cc:564] T 00000000000000000000000000000000 P f759fb68bdac4b619ad7874d0d839b22 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:06:43.222549 23215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df1b6bfb352341c8b9dbef768d7919eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f759fb68bdac4b619ad7874d0d839b22"
I20260929 01:06:43.222849 23214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f759fb68bdac4b619ad7874d0d839b22 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:06:43.223999 23139 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ee7a743fcb249a79ac3a7edeb8055c7 [term 1 FOLLOWER]: Refusing update from remote peer df1b6bfb352341c8b9dbef768d7919eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:06:43.224332 23214 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f759fb68bdac4b619ad7874d0d839b22 [term 1 FOLLOWER]: Refusing update from remote peer df1b6bfb352341c8b9dbef768d7919eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:06:43.224874 23263 consensus_queue.cc:1035] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ee7a743fcb249a79ac3a7edeb8055c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:06:43.225250 23263 consensus_queue.cc:1035] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f759fb68bdac4b619ad7874d0d839b22" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:06:43.226698 23215 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f759fb68bdac4b619ad7874d0d839b22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1b6bfb352341c8b9dbef768d7919eb in term 1.
I20260929 01:06:43.227249 23288 sys_catalog.cc:455] T 00000000000000000000000000000000 P f759fb68bdac4b619ad7874d0d839b22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1b6bfb352341c8b9dbef768d7919eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34219 } } peers { permanent_uuid: "7ee7a743fcb249a79ac3a7edeb8055c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36709 } } peers { permanent_uuid: "f759fb68bdac4b619ad7874d0d839b22" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } } }
I20260929 01:06:43.227392 23288 sys_catalog.cc:458] T 00000000000000000000000000000000 P f759fb68bdac4b619ad7874d0d839b22 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:43.227875 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.211570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53208 (local address 127.0.98.60:37219)
0929 01:06:43.212503 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.212508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.217338 (+ 4830us) server_negotiation.cc:407] Connection header received
0929 01:06:43.217548 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.217552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.217598 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.217650 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.218843 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.219656 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.220886 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.221222 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.221294 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.223496 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:43.223500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:43.223503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:43.223527 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:43.225480 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.226059 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.226064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.226066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.226148 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.226469 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.226472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.226474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.226676 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:43.226845 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.227285 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.227412 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":855}
I20260929 01:06:43.232302 23250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ee7a743fcb249a79ac3a7edeb8055c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df1b6bfb352341c8b9dbef768d7919eb. Latest consensus state: current_term: 1 leader_uuid: "df1b6bfb352341c8b9dbef768d7919eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1b6bfb352341c8b9dbef768d7919eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34219 } } peers { permanent_uuid: "7ee7a743fcb249a79ac3a7edeb8055c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36709 } } peers { permanent_uuid: "f759fb68bdac4b619ad7874d0d839b22" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } } }
I20260929 01:06:43.232429 23250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ee7a743fcb249a79ac3a7edeb8055c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:43.235396 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.216781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32984 (local address 127.0.98.61:36709)
0929 01:06:43.216889 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.216893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.217517 (+ 624us) server_negotiation.cc:407] Connection header received
0929 01:06:43.219641 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.219644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.219693 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.219743 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.221595 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.222255 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.226537 (+ 4282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.226893 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.226986 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.229551 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:43.229555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:43.229558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:43.229584 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:43.233460 (+ 3876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.233985 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.233989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.233991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.234082 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.234406 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.234410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.234412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.234621 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:43.234717 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.235106 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.235228 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 01:06:43.235565 23288 sys_catalog.cc:455] T 00000000000000000000000000000000 P f759fb68bdac4b619ad7874d0d839b22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df1b6bfb352341c8b9dbef768d7919eb. Latest consensus state: current_term: 1 leader_uuid: "df1b6bfb352341c8b9dbef768d7919eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1b6bfb352341c8b9dbef768d7919eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34219 } } peers { permanent_uuid: "7ee7a743fcb249a79ac3a7edeb8055c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36709 } } peers { permanent_uuid: "f759fb68bdac4b619ad7874d0d839b22" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } } }
I20260929 01:06:43.235661 23288 sys_catalog.cc:458] T 00000000000000000000000000000000 P f759fb68bdac4b619ad7874d0d839b22 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:43.236140 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:43.216466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51372 (local address 127.0.98.62:34219)
0929 01:06:43.216571 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.216576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.217605 (+ 1029us) server_negotiation.cc:407] Connection header received
0929 01:06:43.218586 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.218589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.218633 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.218684 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.220671 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.221350 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.227930 (+ 6580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:43.228231 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:43.228304 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:43.229985 (+ 1681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 01:06:43.229989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 01:06:43.229991 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 01:06:43.230020 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 01:06:43.231694 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.234371 (+ 2677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.234375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.234377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.234478 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 01:06:43.234816 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 01:06:43.234819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 01:06:43.234821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 01:06:43.235028 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 01:06:43.235145 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:43.235885 (+ 740us) server_negotiation.cc:299] Negotiation successful
0929 01:06:43.236008 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 01:06:43.237182 23263 sys_catalog.cc:455] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "df1b6bfb352341c8b9dbef768d7919eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1b6bfb352341c8b9dbef768d7919eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34219 } } peers { permanent_uuid: "7ee7a743fcb249a79ac3a7edeb8055c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36709 } } peers { permanent_uuid: "f759fb68bdac4b619ad7874d0d839b22" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } } }
I20260929 01:06:43.237288 23263 sys_catalog.cc:458] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:43.238519 23302 mvcc.cc:205] Tried to move back new op lower bound from 7334477837193023488 to 7334477837155401728. Current Snapshot: MvccSnapshot[applied={T|T < 7334477837193023488}]
I20260929 01:06:43.238658 23288 sys_catalog.cc:455] T 00000000000000000000000000000000 P f759fb68bdac4b619ad7874d0d839b22 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "df1b6bfb352341c8b9dbef768d7919eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1b6bfb352341c8b9dbef768d7919eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34219 } } peers { permanent_uuid: "7ee7a743fcb249a79ac3a7edeb8055c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36709 } } peers { permanent_uuid: "f759fb68bdac4b619ad7874d0d839b22" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } } }
I20260929 01:06:43.238724 23284 catalog_manager.cc:1347] Generated new cluster ID: b0ca1bac188347e387ce2364883e56fe
I20260929 01:06:43.238741 23288 sys_catalog.cc:458] T 00000000000000000000000000000000 P f759fb68bdac4b619ad7874d0d839b22 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:43.238770 23284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 01:06:43.244614 23299 catalog_manager.cc:1259] Loaded cluster ID: b0ca1bac188347e387ce2364883e56fe
I20260929 01:06:43.244709 23299 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f759fb68bdac4b619ad7874d0d839b22: loading cluster ID for follower catalog manager: success
W20260929 01:06:43.245227 23299 catalog_manager.cc:1576] T 00000000000000000000000000000000 P f759fb68bdac4b619ad7874d0d839b22: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260929 01:06:43.245354 23299 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:06:43.247334 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "df1b6bfb352341c8b9dbef768d7919eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1b6bfb352341c8b9dbef768d7919eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34219 } } peers { permanent_uuid: "7ee7a743fcb249a79ac3a7edeb8055c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36709 } } peers { permanent_uuid: "f759fb68bdac4b619ad7874d0d839b22" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } } }
I20260929 01:06:43.247453 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb [sys.catalog]: This master's current role is: LEADER
I20260929 01:06:43.248162 23250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ee7a743fcb249a79ac3a7edeb8055c7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "df1b6bfb352341c8b9dbef768d7919eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df1b6bfb352341c8b9dbef768d7919eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34219 } } peers { permanent_uuid: "7ee7a743fcb249a79ac3a7edeb8055c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36709 } } peers { permanent_uuid: "f759fb68bdac4b619ad7874d0d839b22" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37219 } } }
I20260929 01:06:43.248256 23250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ee7a743fcb249a79ac3a7edeb8055c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:06:43.248458 23300 mvcc.cc:205] Tried to move back new op lower bound from 7334477837193023488 to 7334477837155401728. Current Snapshot: MvccSnapshot[applied={T|T < 7334477837193023488}]
I20260929 01:06:43.262889 23284 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 01:06:43.264086 23284 catalog_manager.cc:1504] Loading token signing keys...
I20260929 01:06:43.275859 23284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df1b6bfb352341c8b9dbef768d7919eb: Generated new TSK 0
I20260929 01:06:43.276683 23284 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 01:06:43.305624 23247 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:43.302555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53216 (local address 127.0.98.60:37219)
0929 01:06:43.302672 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.302676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.304508 (+ 1832us) server_negotiation.cc:407] Connection header received
0929 01:06:43.304698 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.304702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.304750 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.304802 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.305464 (+ 662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53216: BlockingRecv error: recv got EOF from 127.0.0.1:53216 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 01:06:43.307106 23093 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:43.303336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51382 (local address 127.0.98.62:34219)
0929 01:06:43.303445 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.303449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.306167 (+ 2718us) server_negotiation.cc:407] Connection header received
0929 01:06:43.306346 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.306349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.306396 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.306445 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.306994 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51382: BlockingRecv error: recv got EOF from 127.0.0.1:51382 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 01:06:43.308488 23171 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:43.303920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32996 (local address 127.0.98.61:36709)
0929 01:06:43.304029 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.304033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.307471 (+ 3438us) server_negotiation.cc:407] Connection header received
0929 01:06:43.307768 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.307772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.307820 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.307871 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.308377 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32996: BlockingRecv error: recv got EOF from 127.0.0.1:32996 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 01:06:43.312645 23171 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:43.311269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32998 (local address 127.0.98.61:36709)
0929 01:06:43.311372 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.311376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.311778 (+ 402us) server_negotiation.cc:407] Connection header received
0929 01:06:43.311976 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.311979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.312045 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.312091 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.312554 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32998: BlockingRecv error: recv got EOF from 127.0.0.1:32998 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260929 01:06:43.317353 23171 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:43.316005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33010 (local address 127.0.98.61:36709)
0929 01:06:43.316094 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.316097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.316567 (+ 470us) server_negotiation.cc:407] Connection header received
0929 01:06:43.316724 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.316727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.316768 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.316812 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.317259 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33010: BlockingRecv error: recv got EOF from 127.0.0.1:33010 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260929 01:06:43.322583 23171 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:43.321181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33014 (local address 127.0.98.61:36709)
0929 01:06:43.321266 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.321270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.321768 (+ 498us) server_negotiation.cc:407] Connection header received
0929 01:06:43.321931 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.321934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.321976 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.322021 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.322484 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33014: BlockingRecv error: recv got EOF from 127.0.0.1:33014 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260929 01:06:43.329772 23171 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:43.328453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33030 (local address 127.0.98.61:36709)
0929 01:06:43.328579 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.328583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.329001 (+ 418us) server_negotiation.cc:407] Connection header received
0929 01:06:43.329157 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.329159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.329200 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.329243 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.329693 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33030: BlockingRecv error: recv got EOF from 127.0.0.1:33030 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 01:06:43.341033 23171 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:43.339686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33046 (local address 127.0.98.61:36709)
0929 01:06:43.339775 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.339779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.340218 (+ 439us) server_negotiation.cc:407] Connection header received
0929 01:06:43.340407 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.340410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.340453 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.340499 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.340943 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33046: BlockingRecv error: recv got EOF from 127.0.0.1:33046 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260929 01:06:43.360589 23171 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:43.359104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33058 (local address 127.0.98.61:36709)
0929 01:06:43.359217 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.359221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.359712 (+ 491us) server_negotiation.cc:407] Connection header received
0929 01:06:43.359888 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.359892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.359936 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.359984 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.360493 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33058: BlockingRecv error: recv got EOF from 127.0.0.1:33058 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 01:06:43.396889 23171 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:43.395239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33074 (local address 127.0.98.61:36709)
0929 01:06:43.395381 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.395386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.395938 (+ 552us) server_negotiation.cc:407] Connection header received
0929 01:06:43.396133 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.396137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.396188 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.396239 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.396774 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33074: BlockingRecv error: recv got EOF from 127.0.0.1:33074 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 01:06:43.464382 23171 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:43.463350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33088 (local address 127.0.98.61:36709)
0929 01:06:43.463478 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.463482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.463603 (+ 121us) server_negotiation.cc:407] Connection header received
0929 01:06:43.463709 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.463711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.463757 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.463839 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.464225 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33088: BlockingRecv error: recv got EOF from 127.0.0.1:33088 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 01:06:43.595670 23171 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:43.594617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33100 (local address 127.0.98.61:36709)
0929 01:06:43.594779 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.594782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.594870 (+ 88us) server_negotiation.cc:407] Connection header received
0929 01:06:43.595013 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.595015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.595061 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.595132 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.595545 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33100: BlockingRecv error: recv got EOF from 127.0.0.1:33100 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 01:06:43.854956 23171 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 01:06:43.853900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33114 (local address 127.0.98.61:36709)
0929 01:06:43.854156 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:43.854163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:43.854193 (+ 30us) server_negotiation.cc:407] Connection header received
0929 01:06:43.854251 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:43.854254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:43.854301 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:43.854388 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 01:06:43.854787 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33114: BlockingRecv error: recv got EOF from 127.0.0.1:33114 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
I20260929 01:06:44.207476 23261 catalog_manager.cc:1259] Loaded cluster ID: b0ca1bac188347e387ce2364883e56fe
I20260929 01:06:44.207564 23261 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7ee7a743fcb249a79ac3a7edeb8055c7: loading cluster ID for follower catalog manager: success
I20260929 01:06:44.209265 23261 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7ee7a743fcb249a79ac3a7edeb8055c7: acquiring CA information for follower catalog manager: success
I20260929 01:06:44.210433 23261 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7ee7a743fcb249a79ac3a7edeb8055c7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:06:44.247795 23299 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f759fb68bdac4b619ad7874d0d839b22: acquiring CA information for follower catalog manager: success
I20260929 01:06:44.249060 23299 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f759fb68bdac4b619ad7874d0d839b22: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:06:44.372560 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:44.368850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33124 (local address 127.0.98.61:36709)
0929 01:06:44.369087 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:44.369091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:44.369110 (+ 19us) server_negotiation.cc:407] Connection header received
0929 01:06:44.369253 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:44.369255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:44.369380 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:44.369490 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:44.369705 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:44.370272 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:44.371186 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:44.371421 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:44.371505 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:44.371808 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:44.372236 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:44.372336 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:44.372388 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 01:06:44.372433 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":94,"threads_started":1}
I20260929 01:06:44.378898 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:44.375538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53224 (local address 127.0.98.60:37219)
0929 01:06:44.375778 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:44.375782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:44.375794 (+ 12us) server_negotiation.cc:407] Connection header received
0929 01:06:44.375867 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:44.375869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:44.375975 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:44.376067 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:44.376240 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:44.376731 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:44.377530 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:44.377777 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:44.377856 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:44.378082 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:44.378453 (+ 371us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:44.378539 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:44.378593 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 01:06:44.378636 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":79,"threads_started":1}
I20260929 01:06:44.385965 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:44.381675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51390 (local address 127.0.98.62:34219)
0929 01:06:44.382107 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:44.382113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:44.382131 (+ 18us) server_negotiation.cc:407] Connection header received
0929 01:06:44.382234 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:44.382237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:44.382387 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:44.382565 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:44.382870 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:44.383668 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:44.384612 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:44.384846 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:44.384962 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:44.385263 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:44.385500 (+ 237us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:44.385631 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:44.385706 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 01:06:44.385763 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":91,"threads_started":1}
I20260929 01:06:44.386168 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:44.381823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53230 (local address 127.0.98.60:37219)
0929 01:06:44.381940 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:44.381943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:44.382197 (+ 254us) server_negotiation.cc:407] Connection header received
0929 01:06:44.382341 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:44.382346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:44.382394 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:44.382469 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:44.382835 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:44.383584 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:44.384839 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:44.385149 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:44.385213 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:44.385590 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:44.385659 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:44.385710 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:44.385940 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 01:06:44.386046 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 01:06:44.386360 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 01:06:44.382173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33126 (local address 127.0.98.61:36709)
0929 01:06:44.382313 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 01:06:44.382318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 01:06:44.382813 (+ 495us) server_negotiation.cc:407] Connection header received
0929 01:06:44.382986 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 01:06:44.382995 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 01:06:44.383037 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 01:06:44.383116 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 01:06:44.383333 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:44.383949 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:44.385029 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 01:06:44.385360 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:06:44.385426 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 01:06:44.385793 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 01:06:44.385873 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 01:06:44.385921 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 01:06:44.386082 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 01:06:44.386131 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 01:06:44.389113 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 23022
I20260929 01:06:44.395294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 23097
I20260929 01:06:44.401228 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskS2TZOM/build/debug/bin/kudu with pid 23172
2026-09-29T01:06:44Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3039 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6117 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (283507 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST